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Abstract 


A self-stabilizing protocol is one that begins to behave correctly in bounded time, no matter 
what state the protocol is started in. Self-stabilization abstracts the ability of a protocol to 
tolerate arbitrary faults that stop. We investigate the power and applicability of local checking 
and correction for the design of stabilizing network protocols. 

A link subsystem is a pair of neighboring nodes and the two links between them. Intuitively, 
a protocol P is locally checkable if whenever P is in a bad state, some link subsystem is also in 
a bad state. A protocol P is locally correctable if P can be corrected to a good state by locally 
correcting link subsystems. 

We present four general techniques for designing stabilizing protocols. We first show that 
every locally checkable and correctable protocol can be stabilized in time proportional to the 
height of an underlying partial order. Second, we show that every locally checkable protocol 
on a tree can be stabilized in time proportional to the height of the tree. Third, we show that 
every locally checkable protocol can be stabilized in time proportional to the number of network 
nodes. The third result shows that we can dispense with the need for local correctability or the 
need for the underlying topology to be a tree as long as we are willing to pay a higher price 
in stabilization time. Fourth, we show that any deterministic synchronous protocol 7 can be 
converted to an asynchronous, stabilizing version of +. The fourth technique is useful because 
there are network tasks for which a synchronous protocol exists but for which no asynchronous, 
locally checkable solution is known. 

We also present two useful heuristics. The first heuristic, that of removing unexpected packet 
transitions, can often be used to transform a protocol into a locally checkable equivalent. A 
number of existing protocols work in a dynamic network model where links can fail and recover. 
The second heuristic states that locally checkable protocols for dynamic networks can sometimes 
be made locally correctable. The basic idea is to use the link failure and recovery actions of 
the original protocol to locally correct link subsystems. 

Together our techniques cover a broad range of networking tasks. We use our general 
techniques to construct new or improved stabilizing solutions to many specific for Mutual 
Exclusion, Network Resets, Spanning Trees, Topology Update, Min Cost Flows etc. Many 
of our solutions are practical and can be applied to real networks without appreciable loss in 
efficiency. For example, the messages required for local checking can easily be piggybacked on 
the ”keep-alive” traffic sent between neighbors in real networks. 

Our techniques also help in succinctly understanding existing stabilizing protocols. We 


define a special case of local checking called one-way checking. We show that many existing 
protocols implicitly use one-way checking together with two other methods that we call counter 
flushing and timer flushing. 

In the past, papers on stabilization have avoided message passing models of communication 
because of the problems caused by unbounded storage Data links. In a stabilizing setting, 
such links can be initialized with an unbounded number of fictitious packets. Thus almost any 
non-trivial network task is impossible in a stabilizing setting in which the links have unbounded 
storage and the nodes are restricted to be finite state machines. We avoid this problem by using 
the standard asynchronous message passing model of a computer network except that each link 
is what we call a Unit Storage Data Link (UDL) that can store at most one packet. Our UDL 
model can be implemented over real physical channels. Our UDL model also generalizes easily 
to a Bounded Storage Data Link which can store a constant number of packets. 

We introduce a new definition of stabilization in terms of the external behavior of a system. 
The definition allows us to define that an automaton A stabilizes to another automaton B even 
though A and B have different state sets. The definition also allows a clean statement of a 
useful Modularity Theorem. This theorem allows us to prove that a large system is stabilizing 
by proving that each of its pieces is stabilizing. 


Keywords: Self-Stabilization, Fault-Tolerance, Network Protocols, Distributed Algorithms, 
Local Checking and Correction. 
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Chapter 1 


Introduction 


In physics we often talk about systems that stabilize to a good state after initial 
perturbations. For example, a spring eventually stabilizes after being compressed. 
More generally, systems can stabilize to good behavior after an initial perturbation, 
where a behavior is a description of how the state changes with time. For example, 
a missile with a tracking system will continue to move towards its target after it is 
momentarily thrown off course by bad weather. In these examples drawn from physics 
and control theory, the states are continuous variables and the state transitions are 


described by differential equations. 


By contrast, in this thesis we will concentrate on computer systems, and especially 
systems of computers that are interconnected by networks. In such systems, states are 
described by discrete variables and state transitions are described by transition rules, 
often in the form of programs. We will focus on the ability of such computer systems 
to stabilize to “correct behavior” after arbitrary initial perturbation. This property 
was called self-stabilization by Dijkstra |Dij74]. The “self” emphasizes the ability of 


the system to stabilize by itself without manual intervention. 


1.1 A Door Closing Protocol 


A story illustrates the basic idea. Imagine that you live in a house in Alaska in the 
middle of winter. You establish the following protocol (set of rules) for people who 


enter and leave your house. Anybody who leaves or enters the house must shut the 


door after them. If the door is initially shut, and nobody makes a mistake, then the 
door will eventually return to the closed position. Suppose, however, that the door 
is initially open or that somebody forgets to shut the door after they leave. Then 
the door will stay open until somebody passes through again. This can be a problem 
if heating bills are expensive and if several hours can go by before another person 
goes through the door. It is often a good idea to make the door closing protocol 
self-stabilizing. This can be done by adding a spring (or automatic door closer) that 


constantly restores the door to the closed position. 


We can model this situation as a state transition system. To keep things simple, 
let us assume that the door is only used to leave the house, and another door is used 
to enter. The state of the system consists of two Boolean variables, in_threshold and 
door_open. Variable in_threshold is true if and only if a person is in the threshold 
of the door waiting to go out. Variable door_open is true if and only if the door is 
open. We use state transitions called ENTER_THRESHOLD, OPEN_DOOR and LEAVE 
to model the action of a person entering the threshold, opening the door, and leaving 
respectively. We will model errors by allowing the initial values of the two Boolean 


variables to be arbitrary. 


The code for these routines is described in Figure 1.1. The actions are described 
in terms of “preconditions” and “effects”. Preconditions are the enabling conditions 
that must be true before an action can be taken. For instance, we don’t allow the 
OPEN_DOOR action to be taken unless there is a person in the threshold. Effects are 
the results of an action. For instance, the LEAVE action shuts the door. This style of 


description is used throughout the thesis. 


The code also specifies that certain actions take place in time ¢ if they are continu- 
ously enabled. By this we mean that if the preconditions of the action remain true for 
time t, then the action must occur within time ¢. We will use such timing conditions 
throughout the thesis. 


Next, we say that the door closing system is correct if whenever the door is open, 
there is somebody in the threshold. We write this more formally using the predicate 
OK_Door = (If door_open = true then in_threshold = true). For the door closing 
system to be self-stabilizing we want OK_Door to eventually hold regardless of what 
state the system starts in. But it is easy to see that if initially door_open = true and 
in_threshold = false then the predicate OK_Door may never hold. 
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The state of the system consists of two boolean variables 


in_threshold and door_open 


ENTER_THRESHOLD (*user enters the threshold*) 
Preconditions: in_threshold = false 
Effects: in_threshold := true 


OpEeN_Door (*user opens the door*) 
Preconditions: in_threshold = true and door_open = false 


Effects: door_open := true 


LEAVE (*user leaves through open door*) 
Preconditions: in_threshold = true and door_open = true 


Effects: in_threshold := false; door_open := false 


The OpEN_Door and LEAVE actions will occur in time ¢ if 


they are continuously enabled. 


Figure 1.1: Door Closing System 
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RESTORE_Door (*shut the door if there is no user waiting*) 
Preconditions: in_threshold = false and door_open = true 


Effect: door_open := false 


The RestorE_Door action will occur in time t if it is 


continuously enabled. 


Figure 1.2: Extra action that models automatic door closing in a Door Closing System. 


We can model the addition of an automatic door closer using the action RESTORE_DOOR 


shown in Figure 1.2. This door closer works by detecting whether there is anybody in 
the threshold. 


1.2 Self-Stabilization using Domain Restriction 


Consider Figures 1.1 and 1.2 again. Some readers may object that we caused the prob- 


lem by allowing door_open to be an independent variable. Isn’t it possible to hardwire 


relationships between variables to avoid illegal states? In fact, such a technique is 


actually used in a revolving door! 


A revolving door (see Figure 1.3) can be modelled as having three states instead of 


four. A person enters the revolving door, gets into the middle of the door, and finally 


leaves. It is physically impossible to leave the door open and yet there is a way to exit 
through the door. 


This technique, which we will call domain restriction, is a simple but powerful 


method for removing illegal states in computer systems that contain a single shared 


memory. Consider two processes A and B that have access to a common memory 


as shown in the first part of Figure 1.4. Suppose both processes should not run 


concurrently because they can interfere with each other. Thus we would like to provide 


mutual exclusion for the two processes. To achieve this we can pass a token between 
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ye 


STEP 1: ENTERING STEP 2: MIDDLE OF THE DOOR STEP 3: OUT AT LAST! 


Figure 1.3: Exiting through a revolving door. 


A and B. A process can “run” only when it has the token. 


One way to implement this is to use two boolean variables token, and tokeng. The 
token variable for a process is set to true whenever the process has the token. To pass 
the token, Process A sets token, to false and sets tokeng to true. In a self-stabilizing 
setting, however, this is not a good implementation. For instance, the system will 
deadlock if token, = tokeng = false in the initial state. The problem is that we have 
some extra and useless states. The natural solution is to restrict the domain to a single 
bit called turn, such that turn = 1 when A has the token and turn = 0 when B has 


1 


the token. By using domain restriction, * we ensure that any possible state is also a 


legal state. 


In this thesis, we will sometimes use domain restriction to avoid illegal states within 
a single node of a computer network. Domain restriction can be implemented in many 
ways. The most natural way is by restricting the number of bits allocated to a set 
of variables so that every possible value assigned to the bits corresponds to a legal 
assignment of values to each of the variables in the set. Another possibility is to 
modify the code that reads variables so that only values within the specified domain 
are read. Almost all the automata described in this thesis are finite state machines. 
Domain restriction can be performed (for finite state machines) by enumerating the 


legal states and then adding suitable checks to the code. 


Unfortunately, domain restriction cannot solve all problems. Consider the same 


In this example, we are really changing the domain. However, we prefer the term domain restriction. 
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PROCESS PROCESS PROCESS PROCESS 
A B A B 


1. WITH SHARED MEMORY 2. WITHOUT SHARED MEMORY 


Figure 1.4: Token passing among two processes 
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Figure 1.5: A typical mesh Network 


two processes A and B that wish to achieve mutual exclusion. This time, however, 
(see Figure 1.4, Part 2) A and B are at two different nodes of a computer network. 
The only way they can communicate is by sending token messages to each other. Thus 
we cannot use a single turn variable that can be read by both processes. In fact, A 
must have at least two states: a state in which A has the token, and a state in which 
A does not have the token. B must also have two such states. Thus we need at least 


four combined states, of which two are illegal. 


Thus domain restriction at each node cannot prevent illegal combinations across 
nodes. We need other techniques to detect and correct illegal states of a network. It 
should be no surprise that the title of Dijkstra’s pioneering paper on self-stabilization 
[Dij74] was “Self-Stabilization in spite of Distributed Control.” 


1.3 Self-Stabilization in Computer Networks 


In this thesis, we will explore self-stabilization properties for computer networks. A 
computer network consists of nodes that are interconnected by communication chan- 
nels. The network topology (see Figure 1.5) is described by a graph. The vertices of 
the graph represent the nodes and the edges represent the channels. Nodes commu- 
nicate with their neighbors by sending messages along channels. Many real networks 


such as the ARPANET, DECNET and SNA can be modelled in this way. 


A network protocol consists of a program for each network node. Each program 
consists of code and inputs as well as local state. The global state of the network 
consists of the local state of each node as well as the messages on network links. We 
define a catastrophic fault as a fault that arbitrarily corrupts the global network state, 


but not the program code or the inputs from outside the network. 
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Self-stabilization formalizes the following intuitive goal for networks: despite a his- 
tory of catastrophic failures, once catastrophic failures stop, the system should stabilize 
to correct behavior without manual intervention. Thus self-stabilization is an abstrac- 
tion of a strong fault-tolerance property for networks. It is an important property of 


real networks because: 


e Catastrophic faults occur: Most network protocols are resilient to common 
failures such as nodes and link crashes. However, many protocols cannot deal 
with memory corruption. But memory corruption does happen from time to 
time. For example, alpha particles are a common cause of memory corruption. 
It is also hard to prevent a malfunctioning device from sending out an incorrect 
message that carries erroneous state information. The malfunctioning node can 


then crash leaving an incorrect message on a channel. 


e Manual intervention has a high cost: In a large decentralized network, 
restoring the network manually after a failure requires considerable coordination. 
As in the case of the AT&T network, the consequent network shutdown has a 
large dollar cost. Thus even if catastrophic faults occur rarely, (say once a year) 
there is considerable incentive to make network protocols self-stabilizing. In fact, 
a reasonable guideline is what we call Lauck’s Principle [Lau90|. This principle 
states that the network should stabilize preferably before the user notices and 
at least before the user logs a service call. This may seem facetious. However, 
service calls are so expensive that this guideline is sometimes used to set timers 


for self-stabilizing protocols! 


These issues are illustrated by the crash of the original ARPANET protocol ([Ros81] 
[Per83]). The designers used a sequence number to distinguish newer topology updates 
from older ones. Because the set of sequence numbers was finite, they used a circularly 
ordered number space. Hence, it was possible to have three sequence numbers a, b,c 
such that a > b > c > a. The protocol was carefully designed never to enter a state that 
contained the three sequence numbers a, 6, and c. Unfortunately, a malfunctioning 
node injected three such updates into the network and crashed. After this the network 
cycled continuously between the three updates. It took days of detective work [Ros81] 
before the problem was diagnosed. With hindsight, the problem could have been 
avoided by making the protocol self-stabilizing. 
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Self-stabilization is also attractive because a self-stabilizing program does not re- 
quire initialization. The concept of an initial state makes perfect sense for a single 
sequential program. However, for a distributed program an initial state seems to be 
an artificial concept. How was the distributed program placed in such an initial state? 
Did this require another distributed program? Self-stabilization avoids these questions 


by eliminating the need for distributed initialization. 


Probably the most exciting reason for self-stabilization is that can provide a uni- 
form approach towards fault-tolerance, thus leading to simplification as well as strength- 
ening of existing fault-tolerant protocols. This is because self-stabilization can subsume 
such common fault models as link and node failures. However, in order to so the self- 
stabilization recovery mechanisms must be fast enough to provide adequate response 


time for such common failures. 


There appears to be a hierarchy of faults ranging from very rare faults like memory 
corruption (that occur at most once every few days) to fairly common faults like link 
and node crashes (that may occur in the order of minutes) to very common faults like 
bit errors (that may occur every second). Thus it is adequate to recover from memory 
errors in the order of minutes, from link failures in the order of seconds, and from 
bit errors in the order of milliseconds. If the self-stabilization mechanism recovers too 
slowly (say in the order of minutes, as in [Per83]), then it is necessary to have separate, 


faster mechanisms to deal with common failures ([Per83]) like link and node failures. 


On the other hand, if the self-stabilizing mechanisms recover in the order of seconds, 
then there is no need for separate mechanisms to deal with link and node failures. A 
good example of an existing self-stabilizing protocol that meets this criteria is the 
IEEE 802.1 bridge spanning tree protocol described in [Per85]. Clearly reducing the 
number of separate mechanisms leads to simpler protocols. It should be noted that it is 
unlikely that self-stabilization will be efficient enough to subsume the need for all other 
fault-recovery mechanisms; for example, most self-stabilizing protocols will probably 
need some retransmission scheme to deal with messages lost due to bit errors. However, 


the more mechanisms that can be subsumed, the simpler the resulting protocol. 


In this thesis we will investigate methods for designing stabilizing protocols that 
have fast recovery times. Such protocols are not just faster and more fault-tolerant but 
also (by the arguments in the last paragraph) may be simpler than existing protocols. 


The example in Section 1.6.1 should clarify this point. 
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1.4 Criticisms of Self-Stabilization 


Despite the claims of the previous section, there are several peculiar features of the 


self-stabilization model that are often criticized. 


e The model allows network state to be corrupted but not program code. Isn’t 
this distinction artificial? After all programs and state variables are stored as 


bits in memory. 


e The model only deals with catastrophic faults that stop. There are other (e.g., 
Byzantine) models that deal with continuous faults. Aren’t models that allow 


continuous faults preferable? 


e A self-stabilizing program P is only supposed to eventually produce correct be- 
havior. In the interim period, P is allowed to make mistakes. How can we make 


use of a program that can sometimes make mistakes? 


e Most self-stabilizing network protocols require periodic message traffic. Using 
some theoretical measures of message complexity, the message complexity of a 
self-stabilizing protocol is unbounded. Thus a theoretician may question whether 


such protocols are worth the “cost”. 


We deal with each criticism in turn: 


1.4.1 Distinction between Program Code and State 


Program code can be protected against arbitrary corruption of memory by redundancy 
since code is rarely modified. Some static input (such as node IDs) can also be pro- 
tected in this way and can be considered to be part of the program. Some changing 
input (such as the list of neighboring nodes in a network) can be protected by requiring 
that such input be the output of another self-stabilizing protocol. On the other hand, 
the state of a program is constantly being updated and it is not clear how one can 
prevent illegal operations on the memory by using checksums. It is even harder to 


prevent a malfunctioning node from sending out incorrect messages. 
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1.4.2 Faults that Stop versus Faults that Continue 


In the Byzantine |LSP82] fault model, some fraction of faulty nodes can continuously 
exhibit arbitrary faulty behavior. By allowing continuous faults, the Byzantine fault 
model appears to be stronger than the self-stabilization model. However, network 
protocols with Byzantine robustness |Per88] are expensive because they require large 
amounts of redundancy in storage, processing, and message traffic. On the other hand, 
it is possible to make many protocols self-stabilizing with a small cost in extra message 


traffic and node processing. 


In Byzantine models, only a fraction of nodes are allowed to exhibit arbitrary 
behavior. In the self-stabilization model, all nodes are permitted to start with arbitrary 
initial states. Thus, neither model subsumes the other. In theory, there is no reason 
why a protocol cannot be robust against both Byzantine failures and arbitrary initial 


states. 


Assuming that faults stop in the self-stabilization model is only a modelling arti- 
fice. In practice, we only need faults to stop for a period long enough for the protocol 
to stabilize. Thus the self-stabilization model is especially appropriate for handling 


transient errors. 


1.4.38 Permitting Initial Errors 


A distributed database program cannot tolerate errors that may, for instance, wrongly 
credit an account with a million dollars! However, for most of the network protocols 
considered in this thesis, errors are not as critical. An example is a network protocol 
that computes routes between nodes. The nice thing about a routing protocol is that 
even if the network is completely fouled up, the worst thing that can happen is that 
network traffic stops for a while. Most of the stabilizing protocols described in this 
thesis are used for routing, scheduling, and resource allocation tasks. For such tasks, 


initial errors only result in a temporary loss of service. 


1.4.4 Periodic Message Sending in the Self-stabilization Model 


It is easy to show that any non-trivial self-stabilizing network protocol must send mes- 


sages periodically. Periodic sending of messages may seem extremely ugly. However, 
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in real networks, each node periodically sends control messages to its neighbors to de- 
tect whether the neighbor is alive. For many of the protocols described in this thesis, 
the periodic message sending required for stabilization can be piggybacked on such 


“keep-alive” message traffic without appreciable loss of efficiency. 


In a real implementation, periodic message sending is controlled by timers in order 
to keep the overhead bounded. We will not model these timers explicitly. A temer 
can be implemented in a self-stabilizing fashion as long as the hardware clock in every 
node that’s up continues to function. For instance, we can implement a timer using a 
counter that is incremented every time the hardware clock ticks. When the counter 
reaches its maximum value, the sending of a message is enabled, and the counter is 
reset to 0. Assuming that the hardware clock continues to tick is not at all restrictive. 


For most computers, if the hardware clock stops, the node has effectively crashed! 


1.5 Brief History of Self-Stabilization 


Self-stabilization was introduced by Dijkstra in a seminal paper [Dij74]. In Dijkstra’s 
model, a network protocol is modelled using a graph. The nodes of the graph contain 
finite state machines. The protocol is asynchronous, and the asynchrony is modelled 
by an adversarial scheduler called a “demon”. At each stage, the demon is allowed to 
choose an arbitrary node in the graph to make a move. In a single move, a node is 
allowed to read the state of its neighbors, compute, and then possibly change its state. 


In this setting, Dijkstra described three self-stabilizing mutual exclusion protocols. 


After Dijkstra’s initial paper, work on self-stabilization languished for many years. 
However, in this period, at least three researchers recognized the importance of the con- 
cept, and championed its cause. Gouda and his co-workers at the University of Texas 
produced a number of papers (e.g., [BGW87],/GM90], [AG90]) in this area. Lam- 
port’s PODC address ({Lam84]) was probably responsible for awakening the interest 
of the theory community in self-stabilization. Independently, Tony Lauck, |Lau90], 
who is responsible for the architecture of DECNET, recognized the applicability of 
self-stabilization to real networks. At his insistence, self-stabilization was added as a 
requirement ([Per83]) for many DECNET protocols. 


After Lamport’s PODC address, a number of papers began to appear in this area. 


The contributions of these papers fall into three categories: refinements of Dijkstra’s 
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model, solutions to specific tasks, and one general technique. 


1.5.1 Refinements of Dijkstra’s model 


In Dijkstra’s model, a node is allowed to read the state of all its neighbors and change 
its own state, all in one move. This level of atomicity is hard to achieve in a real 
network. [BGW87] suggest a model in which at each step, the demon can allow 
an arbitrary subset of nodes to make a move. Later [DIM90] introduced a model 
in which a node communicates with its neighbors by reading or writing to certain 
shared registers. Also, in their model, reading and writing of the shared register 
(and local computation) are separate atomic steps that can be arbitrarily interleaved. 
Other papers [Per83, AB89, GM90, KP90] model communication between nodes by 


the explicit sending of messages. 


In Dijkstra’s model, one node in the graph is assumed to be a “leader” in order to 
break symmetry. Dijkstra observed that some form of symmetry breaking is required 
for self-stabilizing mutual exclusion. Later models introduced other forms of symmetry 
breaking. In [Per83, AKY90], each node is assumed to have a distinct ID. [IJ90| 
introduced the use of randomization. Finally, most papers in this area assume the 
model is completely asynchronous. No assumptions are made about how long it takes 
for actions to be performed. By contrast, [Per83, Per85] assume upper bounds on 


message delivery and node processing times. 


1.5.2 Existing solutions for Specific Tasks 


Dijkstra’s paper concentrated on the task of mutual exclusion on rings. Subsequent 
papers (e.g., [BP89, DIM90, 1J90]) continued to work on self-stabilizing mutual exclu- 


sion, but in different models. Solutions to other tasks have also appeared. 


[Per83, SG89] describe self-stabilizing routing protocols to compute shortest path 
routes between every pair of nodes in a network. [Per85, AG90, AKY90] describe 
self-stabilizing protocols to compute a spanning tree in a network. [AB89, GM90] 
show how to establish reliable communication between a pair of nodes over a physical 
channel. A reset protocol is a protocol that can be used to “reset” a network to a 


prespecified initial state; a snapshot protocol can be used to find a consistent global 
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state of a network. [AG90, KP90] describe self-stabilizing snapshot and reset protocols. 


[Spi88a] describes a self-stabilizing virtual circuit protocol. 


1.5.3 Existing General Technique 


While many fundamental problems have been tackled, there is a lack of general meth- 
ods. We know of only one general technique other than the work described in this 
thesis. Katz and Perry [KP90] show how to stabilize a large class of distributed algo- 
rithms by centralized checking and correction at a leader. The main technical difficulty 
in this approach is finding a self-stabilizing method to do checking and correction. For 
this purpose, [KP90] invented a self-stabilizing snapshot protocol. However, the need 
for centralized checking makes the performance of this approach rather poor as we see 


in the next section. 


1.6 Local Checking and Correction: A Preview 


The major theme of this thesis is the design of new and efficient general methods 
for making protocols self-stabilizing. All our methods are based on what we call 
local checking. Unlike [KP90], our methods are efficient because checking is local and 


decentralized. In this section, we give a preview of our ideas. 


1.6.1 Example of Checking and Correcting on a Single Link 
Subsystem 


To make the notion of local checking and correcting more concrete we quickly describe 
an example of a protocol that works between two nodes, a sender node and a receiver 
node (Figure 1.6) that are connected by two unidirectional links. The sender sends 
messages to the receiver who buffers the messages in a finite sized queue. Any message 
that arrives when the queue is full is dropped. A simple credit-based scheme can be 


used to prevent messages being dropped during normal operation. 


The sender (Figure 1.6) keeps a credit register which stores the current credits 


available to the sender. Initially, assume that the receiver queue is empty and that the 
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Figure 1.6: Credit Based Flow Control between a Sender and a Receiver 


sender’s credit register is equal to the size of the receiver queue (say Maz). The sender 
only sends a message when the credit register is non-zero; after sending a message, the 
sender decrements its credit register by one. When the receiver removes a message 
from its queue, the receiver sends a CREDIT message back to the sender. When the 
sender receives such a CREDIT message, the sender increments its credit register by 


one. 


It is easy to see that after proper initialization and assuming that no errors occur, 
no messages will be dropped. Under such conditions, the following condition (which 
we will later call a Local Predicate) holds at every instant. If at any instant we denote 
(see Figure 1.6) the value of the credit register by C’, the number of messages in flight 
by M, the number of messages in the queue as Q and the number of credits in flight 
by CR, then it must be true that: C+ M+Q+CR= Maz. Intuitively, this can be 
seen by analogy to two banks that only transfer money between each other; assuming 
no errors, the total amount of “money” (credits plus messages) in and between the 
two banks must be conserved. This local predicate ensures that there is always room 


in the queue for the messages in flight since M+ Q < Maz. 


Unfortunately, this simple credit based scheme runs into trouble if the system is 
either improperly initialized or there are errors on the link. Link errors can result 
in lost or even (less likely) added credits. Credit loss can result in slowing down the 
sender and possibly even deadlock; credit addition can lead to continuous dropping of 


messages. 


We can make this protocol self-stabilizing by superimposing a periodic check- 
ing/correcting process (see Figure 1.7) on the original protocol. This process is trig- 


gered by a timer at the sender every few seconds. To initiate a checking phase, the 
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Figure 1.7: A Single Phase of Checking/Correction using Snapshots/Resets 


sender (see Figure 1.7) sends a Snapshot ([CL85]) request message to the receiver. 
While checking, the sender also stops sending any messages on the link. When the 
receiver receives such a message, the receiver sends back a response containing the 
number of messages in its queue (Q) at the instant it sent the response. When the 
sender gets the response, the sender checks whether Q + C = Maz, where C is the 
value of the credit register at the instant the response is received. If this condition is 


false, the sender infers that the local predicate is violated and initiates a reset phase. 


To initiate a reset phase, the sender (see Figure 1.7) sends a Reset request message 
to the receiver. As in checking, the sender also stops sending any messages on the 
link until it gets a response.” When the receiver receives such a request, the receiver 
empties its queue and sends back a response. When the sender gets the response, the 
sender reinitializes its credit register to Maz. In other words, the receiver reinitializes 
its local state on sending the response and the sender reinitializes its local state on 
receiving the response. Its not hard to see that if no errors occur during the reset 


phase, the local predicate will hold at the end of the reset phase. 


Several optimizations can be added to this basic scheme. For example, it is possible 
to avoid having the sender stop sending messages during a checking phase by keeping 
track of some extra state variables. For this particular example, it is also possible to 
avoid a separate reset phase; instead when the sender receives a snapshot response, 


the sender can locally correct the credit register to account for any discrepancy. 


2Chapter 5 contains details of how this protocol deals with lost request and responses. 
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We can now illustrate the point made earlier that self-stabilization can subsume 
the need for other fault-tolerance mechanisms. If the checking/correction procedure 
is activated fairly often (doing it once every second on a high speed link requires 
negligible overhead), then there is no need for separate mechanisms when either the 
sender or receiver nodes or the two links crash and recover. For a crash and recovery 
we do nothing special. Clearly the local predicate can be violated by such actions. 
However, after the next checking and correction phase the flow control scheme will 
begin working correctly. In the interim, messages may be lost but this is comparable 
to the time most protocols take to reinitialize after a link recovers; during this period 
the protocol is not providing service to the user. Thus the final scheme is both simple 


and fault-tolerant. 


This fault-tolerant credit based scheme was proposed by us (for use on high speed 
links) at Digital Equipment Corporation ([CSV89]). Recently, we proposed a variant 
of this scheme for hop-by-hop flow control on ATM? at the ATM Forum meeting in 


Aug 1993. Local checking and correction is practical! 


1.6.2 Extending the Idea to a General Network 


Briefly, the rest of this thesis can be described as an extension (of the simple link 
checking and correction scheme described in the last subsection) to general network 


protocols. 


So consider a network as shown in Figure 1.5. Recall that in the method of |[KP90| 
there is a leader that periodically checks the network. If the leader discovers that the 
network is in an illegal state, the leader corrects the network by resetting it into a 
good state. Intuitively, centralized checking and correction is slow. It also has high 


message complexity. 


Instead, we divide the network into a number of overlapping link subsystems as 
shown in Figure 1.8. A link subsystem consists of a pair of neighboring nodes and the 
channels between them. We wish to replace the global, centralized checking of [KP90] 
with local, decentralized checking. The intent, of course, is to allow each link subsystem 


to be checked in parallel. This results in faster stabilization. 


3ATM stands for Asynchronous Transfer Mode. In ATM, messages are fixed sized “cells”. There can 


also be multiple “circuits” per link each of which must be independently flow-controlled and checked. 


25 


SUBSYSTEM S i 


Figure 1.8: An Example of Two Overlapping Link Subsystems in a Network 


We describe sufficient conditions under which these methods can be applied. In- 
tuitively, a network protocol is locally checkable if whenever the protocol is in a bad 
state, some link subsystem is also in a bad state. Thus if the protocol is in a bad 
state, some link subsystem will be able to detect this fact locally. As in [KP90], we 
can correct a locally checkable protocol by doing (what we call) global correction of the 
network. However, in some cases we can do even better if the protocol is also locally 


correctable. 


Intuitively, a network protocol is locally correctable if the network can be corrected 
to a good state by each link subsystem independently correcting itself to a good state. 
Clearly, this is non-trivial because link subsystems overlap (see Figure 1.8) at nodes. 
In the figure, the correction of link subsystem 5, may cause subsystem 5S, to become 


incorrect. 


1.6.3 Examples of Local Checking and Correction 


We will go through three simple examples to make these notions clearer. The first 
example is not locally checkable, the second is locally checkable but does not appear 


to be locally correctable, and the third is both locally checkable and locally correctable. 


For the first example, consider a token passing protocol in a line graph as shown 
in Figure 1.9. The line is oriented such that A is at the leftmost end and X is at 
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Figure 1.9: Token Passing in an Oriented Line Graph is not Locally Checkable 


Red Blue Blue Orange 


Figure 1.10: Coloring a Cycle is Locally Checkable but not Locally Correctable 


the rightmost end. In normal operation, a single token is passed from left to right 
(i.e.. from A to X), and then back from right to left (i.e., from X back to A). Thus 
each node in the line will receive the token periodically. This protocol is not locally 
checkable if the graph has at least 3 nodes. Consider a typical link subsystem, for 
example the subsystem between neighbors B and C’ in Figure 1.9. Clearly, in normal 
operation it is possible for there to be no token at either B, C’, or the channels between 
them. Thus having no tokens in a link subsystem is a legal state of a link subsystem. 
But this means that if there is no token in the entire network, no subsystem can detect 


this fact locally. Remember that subsystem checking is not coordinated. 


For the second example, consider a protocol that colors the nodes of a cycle as 
shown in Figure 1.10. We require that the color of each node be either red, blue 
or green. We also require that the color of each node be different from that of its 
neighbors. Assume that in one atomic step a node can read the state of its neighbors 


and change its own state. However, steps of nodes can be arbitrarily interleaved. 


Then the protocol is locally checkable. Suppose that node A has the same color as 
a neighbor B. Then, this can be detected within the link subsystem containing A and 
B. 


However, it is not clear how to make this protocol locally correctable. Suppose 
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that all nodes are initially red. Then by the symmetry of the initial state, it appears 
that local corrections (or any corrections) are insufficient to correct the system to a 
good state. We could break symmetry with randomization. However, in this thesis we 


will only consider deterministic local correction procedures. 


Consider the same problem of coloring the nodes of a graph, except that the graph 
is an oriented line graph as shown in Figure 1.9. Then the protocol is locally checkable 
and locally correctable. Suppose that that two nodes in a link subsystem (say B and 
C’) have the same color. Then to correct the link subsystem, the color of the right node 
(i.e., C’) is changed to any legal color different from the color of left node (i.e., B). 
Assume that correction actions occur in bounded time after they are enabled. Then 
within bounded time, node B will have a color different from that of A and will never 
change its color from this point on. Then within bounded time after node B’s color 
stabilizes, node C’ will have a color different from that of B and will never change 
its color from this point on. By induction, we can show that all nodes are colored 


correctly in bounded time. 


1.6.4 Why Local Checking is Useful 


It is perhaps surprising that a number of useful network protocols are both locally 
checkable and locally correctable. In subsequent chapters we will describe locally 
checkable and correctable protocols for mutual exclusion, network resets, and end- 
to-end communication across an unreliable network. It may seem from the simple 
examples that our method is confined to acyclic graphs; this is not true: both the end- 
to-end and reset protocols work on arbitrary topologies. It also appears (see Chapter 
7) that other existing protocols that work in dynamic networks (in which the topology 
can change due to link failures and recovery) are locally correctable. Protocols that 


are both locally checkable and correctable can be stabilized very quickly. 


Protocols that are locally checkable but work on a tree topology can be stabilized 
in time proportional to the height of the tree. Thus we can remove the need for 
local correctability if the underlying topology is a tree. Another way to remove the 
need for local correctability (without restricting the topology to a tree) is to pay 
a price in stabilization time. Protocols that are locally checkable but not locally 
correctable can be made self-stabilizing by doing global correction using the network 


reset protocol developed in this thesis. The price for using global correction is that 
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the stabilization time now becomes proportional to the number of nodes. We describe 
stabilizing spanning tree and topology update protocols that use local checking and 


global correction. 


We will also describe two compilers that can compile any deterministic synchronous 
protocol 7 into a self-stabilizing asynchronous version of 7. The first compiler is 
stabilized by using local checking and correction, while the second compiler is stabilized 
using local checking and global correction. The significance of the compilers is that 
there are some network tasks (for example, computing a minimal spanning tree) for 
which a synchronous protocol exists but for which no locally checkable solution is 


known. Hence the compilers extend the range of our general techniques. 


Thus while local checking cannot be used to solve every problem, there are a large 
number of useful protocols that can be efficiently stabilized using this notion. There 


are several benefits to this approach: 


e The resulting protocols are efficient and stabilize quickly. 


e The approach allows us to understand how to design self-stabilizing protocols 
in a systematic fashion. In fact, we will show that some existing self-stabilizing 


protocols can easily be understood in this framework. 


e The approach allows us to prove self-stabilization properties of protocols in a 
modular way. This is because we limit ourselves to proving properties of link 


subsystems instead of arguing about global states. 


e As a side benefit, local checking provides a useful debugging tool. Recall that 
each link subsystem periodically checks whether the subsystem is in a good 
state. Thus any violations can be logged for further examination. In a trial 
implementation of our reset procedure on the Autonet [MAM*90], local checking 
discovered bugs in the protocol code. In the same vein, local checking can provide 


a record of catastrophic, transient faults that are otherwise hard to detect. 


1.7 Thesis Organization 


The thesis is organized into three major parts as illustrated in Figure 1.11. The first 


part consists of three chapters on basic definitions and examples. The second part 
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Figure 1.11: Thesis Organization 


contains three chapters on local checking and local correction. The third part consists 
of two chapters on local checking and global correction. The final chapter presents our 
conclusions and contains a list of open questions. There are also several appendices. 
The first appendix is a list of frequently used notation. Next, there are appendices 
containing some details of proofs that were omitted in the main text for clarity. Finally, 
there is an index of commonly used terms and definitions. Figure 1.11 also summarizes 


the major results of the thesis. 


We now describe each of the major parts in more detail below. 


1.7.1 Basic Definitions and Examples: Chapters 2 - 4 


Chapter 2 describes our model of computation, a variant of the timed Input/Output 
automata model ([MMT91], [LT89]. The model is basically a state machine model 
except that transitions are labelled with action names. By separating actions into 
internal and external actions, it is possible to define the correctness of an automaton 


in terms of its external behavior, where a behavior is a sequence of external actions. 
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Chapter 3 contains our definitions of stabilization. Roughly, we say that an au- 
tomaton A stabilizes to some target set P of behaviors if every behavior of A has a 
suffix that isin P. The intuition is that the behaviors of A eventually begin to “look 
like” the behaviors in P. The actual definitions are slightly more complex in order 
to define what it means to stabilize in bounded time. Our behavior definitions are 
in contrast to previous definitions (e.g., [KP90] which are in terms of the states and 
executions of a system. We do have a definition of stabilization that corresponds to the 
standard definition; we use the behavior definition for specification and the standard 


definition for proofs. 


Chapter 3 also contains our first important result. This is a Modularity Theorem 
that allows us to prove facts about the stabilization of a large system by proving facts 
about the stabilization of the system components. The theorem formalizes a “building 
block” approach to designing stabilizing protocols that we use throughout the thesis. 
Chapter 3 also describes a technique for proving stabilization properties. Chapter 3 is 


joint work with Nancy Lynch. 


Chapter 4 contains a quick example of local checking and correction in the simplified 
shared memory model introduced by Dijkstra in [Dij74]. Because Dijkstra’s model is 
so simple, it allows us to strip away extraneous detail and focus on the main ideas 
behind local checking and correction. However, readers who wish to concentrate on 
results for more realistic network models should skip Chapter 4 and go directly to 
Chapter 5. From Chapter 5 onwards, we use a network model suitable for modelling 
real networks. Chapter 4 is based on work done by the author. Independently, Anish 
Arora and Mohamed Gouda from the University of Texas at Austin have obtained 


similar results. Joint publication is planned. 


1.7.2 Local Checking and Correction: Chapters 5 - 7 


Chapter 5 begins by introducing our network model. Our network model is basically 
the standard asynchronous message passing model except for one important twist: 
each link 1s restricted to store at most one packet at a time. We argue that bounded 
storage link models are essential in a stabilizing context. We also argue our network 
model can be easily implemented in real networks. The network model of Chapter 5 


is used for the reset of the thesis. 
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The concept of local checking and correction was introduced in [APV91b], and 
is joint work with Baruch Awerbuch and Boaz Patt. While these concepts are used 
throughout the thesis, [APV91b] did not present a formal description of the method. 
Instead [AP V91b] described the method informally and showed how it could be used 
to stabilize two important protocols, one for end-to-end message delivery and one for 


network reset. 


Chapter 5 gives a formal basis for the method of local checking and correction in 
message passing systems. The chapter contains formal definitions of local checkability 
and local correctability in a network model. These definitions are used to state the 
main result of the chapter, the Local Correction Theorem. This theorem shows that 
any locally checkable and correctable protocol can be transformed into an equivalent 
stabilizing protocol. The stabilization time of the resulting system is proportional to 
the height of a certain partial order that is used in the definition of local correctability. 


Chapter 5 is joint work with Nancy Lynch. 


Chapter 6 applies the method of local checking to a simple mutual exclusion proto- 
col. Chapter 6 also contains an important result, the Tree Correction Theorem. This 
theorem states that any locally checkable protocol on a tree can be efficiently stabilized 
in time proportional to the height of the tree. In other words, if the underlying topol- 
ogy is a tree we can dispense with the need for local correctability. The proof of this 
theorem is only sketched because we prove a corresponding tree correction theorem 


for shared memory systems in Chapter 4. 


Chapter 7 links the second and third parts of the thesis by describing a stabilizing 
network reset protocol. Intuitively, a network reset protocol is a protocol that can 
be used by some other protocol P in order to restore P to a good state. Protocol 
P is given interfaces to make reset requests; the network reset protocol responds by 
providing reset signals at each network node. If each node (that implements P) locally 
initializes its state at the instant it receives a signal, then P will be restored to a good 


state. 


In order to use such a network reset protocol as a tool for building other stabilizing 
protocols (as we do in the third part of the thesis) the network reset must itself be 
stabilizing. Chapter 7 applies the method of local checking and correction to create a 
stabilizing network reset protocol as described in [APV91b]. Chapter 7 is joint work 
with Baruch Awerbuch and Boaz Patt. 
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Chapter 7 also explores an interesting heuristic connection between locally cor- 
rectable protocols and protocols that work in dynamic networks where links can fail 
and recover. The heuristic states that locally checkable protocols for dynamic networks 
can sometimes be made locally correctable. The basic idea is to use the link failure 
and recovery actions of the original protocol to locally correct link subsystems. This 


heuristic is the key to the proof of local correctability for our network reset protocol. 


1.7.38 Local Checking and Global Correction: Chapters 8 - 9 


The last part of the thesis contains two applications of global correction. 


In Chapter 8 we prove another major result, the Global Correction theorem. This 
theorem states that any locally checkable protocol can be stabilized in time propor- 
tional to the number of network nodes. The Global Correction theorem shows that 
we can dispense with the need for local correctability and the need for the underlying 
topology to be a tree as long as we are willing to pay a higher price in stabilization 
time. The height of the underlying partial order in the local correction method and the 
height of the tree in in the tree correction method are typically smaller than the num- 
ber of network nodes. Thus it pays to use local correction or tree correction wherever 


possible. 


We present stabilizing protocols for computing a spanning tree and solving the 
topology update problem as examples of Global Correction. The spanning tree and 
topology update protocols are based on joint work with Baruch Awerbuch and Boaz 
Patt that is also described in [APV91la]. The protocols in Chapter 7 and 8 are both 


efficient and practical and can be applied to real networks. 


Chapter 9 develops two compilers that can convert any synchronous protocol z into 
a self-stabilizing asynchronous version of 7. The main compiler, the Resynchronizer, 
works by first applying the synchronizer protocol of [Awe85] to create an asynchronous 
version of 7. Next we use global correction to make the resulting protocol stabilizing. 
This can be done by using a stabilizing reset protocol to periodically restart an asyn- 
chronous version of protocol 7. The proof of the current version of the Resynchronizer 
protocol is incomplete. However, we have a proof of a much more complicated version 
of the Resynchronizer protocol that was originally reported in [AV91]. The construc- 


tion in this thesis is much simpler than our original construction in [AV91] but its 
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proof is incomplete. Thus chapter 9 is best regarded as a set of useful ideas that need 


polishing. Chapter 9 is joint work with Baruch Awerbuch 


1.8 Reading the Thesis 


Most chapters and long sections contain a roadmap at the start that explains the 
organization of the chapter or section. Similarly most chapters end with a summary 
of the important ideas in the chapter. Since it is easy to forget a piece of notation or 


a definition, the reader may also wish to consult the notation appendix and the index. 


Whenever the proof of a theorem or lemma is too long, we give an intuitive expla- 
nation of why the theorem or lemma works in the main text, and provide more details 


later or in the appendix. On a first reading, the reader is advised to skip the details. 


We believe that self-stabilization is useful and practical and hope that systems 
readers can also read this thesis. Chapter 2 is written to help readers unfamiliar 
with formal methods to get comfortable with the formalism we use. A systems reader 
wishing to get a quick summary of the results can read the introduction, summary and 
main theorems in each chapter. Once the reader gets comfortable with our method of 
describing protocols it should also be easy to read the actual code of the protocols. The 
complicated (and important) pieces of code are heavily commented and are preceded 


by informal descriptions. 
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Chapter 2 


The I/O Automaton Model 


A formal description of an algorithm is a precise and unambiguous description of the 
algorithm. Formal descriptions of sequential algorithms have proved to be useful. 
Distributed algorithms are more complicated than sequential algorithms because they 
have to deal with parallelism, asynchrony, and fault-tolerance. Thus we will often give 


a formal presentation of the protocols in this thesis. 


By describing algorithms formally, we hope to describe them precisely so as to 
avoid ambiguity and to permit careful proofs of correctness. However, it is often hard 
to do so and yet convey the important ideas. We will try to combat this by providing 


intuitive explanations along with formal descriptions. 


To describe algorithms precisely, we use an underlying mathematical model. The 
idea is that after we model a real-life distributed algorithm, we can study the algorithm 
purely in terms of its mathematical properties. Despite this, we will often return to 


what these mathematical symbols represent. 


In this chapter we will describe the computation model used to describe the dis- 
tributed algorithms in this thesis. The first section of this chapter is an intuitive 
introduction to the I/O automaton model. This section is written for readers unfa- 
miliar with the I/O automaton model. The second section of the chapter contains a 
formal description of the variant of the I/O automaton model that we use in the rest 
of the thesis. Readers already familiar with the I/O automaton model may wish to 


only read the formal description in Section 2.2. 
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2.1 The I/O Automaton Model 


Our model of computation is a variant of the timed Input/Output automaton model 
[MMT], which in turn is based on the Input/Output automaton model of Lynch and 
Tuttle [LT89]. We will omit the word “timed” in what follows. For instance, we will 


refer to a timed I/O automaton simply as an automaton. 


2.1.1 Why use the I/O Automaton Model? 


We wish to model systems of processes that compute but also communicate with 
other processes. The processes do not have a common clock and so communication is 
asynchronous. A sequential algorithm computes some function of its input and then 
halts. By contrast, our processes can continuously receive input from and react to 


their environment. 


Consider the example of a token passing ring. Such rings are the basis of a number 
of local area networks that interconnect computers in offices and on college campuses. 
A token passing ring consists of a number of processes, say 0 to n — 1, connected 
together in a ring. To prevent more than one process from transmitting at the same 
time, a token packet is passed from process to process. A process can transmit only 
when it has a token. A process passes its token to its clockwise neighbor a bounded 
time after it finishes transmitting. It is easy to see that the system works correctly if 


there is exactly one token in the system initially. 


The simplest model of the token passing system is a big state machine. Each node is 
a state machine that sends and receives packets and so are the channels between nodes. 
The state of a channel is the sequence of packets stored in the channel. Unfortunately, 
such monolithic models often do not have a property we call compositionality. A model 
is said to be compositional if we can infer the behavior of the system from the behavior 


of its components. This allows modular specification and modular proofs. 


The I/O automaton model is essentially a state machine model. However, it has a 


few extra features that make it a compositional model. 
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i i 1 
INTERFACE TO A PROCESS INTERFACE TO A CHANNEL 


Figure 2.1: The Interface to Process i and the Channel between Process i and Process 1+ 1 in a token 


passing system 
2.1.2. Four Important Features of the I/O Automaton Model 


In the I/O automaton model, both systems and processes are modelled by an I/O 
automaton. An I/O automaton is an automaton (i.e., a state machine) with the 


following additional features. 


First, all state transitions of the automaton are labelled with names that are known 
as actions. Further, all actions are classified into three categories: input, output, 
and internal. Intuitively, input actions are actions caused by the external world or 
environment and to which the automaton must respond. Output actions are actions 
caused by the automaton and to which the environment must respond. Finally, internal 
actions are transitions that are neither input or output actions: such actions only 


change the internal state of the automaton. 


As an example, Figure 2.1 shows a single process in the token passing system, say 
Process 1. Of course, Process 1 must have interfaces to send and receive packets. We 
can model these interfaces using an input action RECEIVE;(p) to receive a packet p 
and an output action SEND,;(p) to send a packet p. In the figure, we have not shown 
any internal actions. Figure 2.1 also shows the external interface to a channel between 
Process i and Processi+1. Notice that SEND;(p) is an input action and RECEIVE;+1(p) 
is an output action for this channel. Intuitively, this corresponds to the fact that when 
Process 1 sends a packet as an output action, that packet must simultaneously be 


stored in the channel using a channel input action. 


The classification of actions allows a simple scheme for “plugging” together au- 
tomata so that they can can communicate. The formal name for this scheme is compo- 


sition. Composition is based on an idea we have already alluded to. When automata 
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Figure 2.2: The token passing system formed by composition of processes and channels 


are composed, the output actions of the automata are identified with input actions 
of other automata that share the same name. As an example, suppose we compose 
Process 1 with the channel between Process i and Processi+1. Then the output action 
SEND,(p) of Process i gets identified with the input action SEND,;(p) of the channel. 
When we run the new composed automaton, whenever Process i performs a SEND;(p) 
action, the channel will simultaneously perform a SEND;(p) action. Thus the second 
feature of the I/O automaton model is that automata communicate by simultaneous 


performance of shared actions. 


Continuing our example, we can compose the automata for Process 1,0 <i <n-—1 
and the channels between them to form a new automaton that represents the token 
passing system. This is shown in Figure 2.2. We assume that all arithmetic on process 


indices is mod n. 


Suppose an automaton A wishes to perform an output action 7 that is also an input 
action of another automaton B. Some models allow automaton B to block its inputs in 
certain states. If B can block input actions, then A must somehow “handshake” with 
B before A can perform action 7. This in turn implies that action z is jointly controlled 
by A and B. In the I/O automaton model, things are much simpler because of a third 
feature of the model. Input actions are enabled in every state of an automaton. Thus 
there is no need for handshaking, and an action is controlled solely by the originator 


of the action. 


The assumption that inputs cannot be blocked is extremely natural for the message 
passing systems studied in this thesis. In real message passing systems, a process must 
be prepared to receive packets in any state. Of course, a process may choose to drop 
a packet when it receives it. On top of this basic model, processes may choose to 


implement a flow control scheme to prevent senders from sending to receivers when 
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the receivers are unable to process packets. However, such “flow control” is not part 


of the basic I/O automaton model. 


Output and internal actions of an automaton A are under the control of A; hence 
they are also called locally controlled actions. Typically, we need to specify certain 
“liveness” guarantees on the performance of locally controlled actions. For example, 
in the token passing system we need to specify that a node holds the token for a 
bounded amount of time. We specify such guarantees using a fourth feature of timed 
automata. The locally controlled actions of an automaton are partioned into a number 
of equivalence classes. Each class c in this partition has a time t, which represents an 


upper bound on the performance of an action in class c. 


Intuitively, each class represents the set of actions under the control of one system 
component. The automaton will guarantee “fair turns” to the enabled actions in each 
class. Suppose some action of class c is enabled at time ¢. One can think of the 
automaton as having a scheduler that checks every class at time periods of at least 
t,. If some action of class c is enabled when the scheduler checks the class, then some 
action in the class is performed. More precisely, we require that either some action 
of class c will be performed by time ¢ + t,, or no action of class c is enabled in some 
state that occurs before time ¢ + ¢,. Returning to our example, we specify that each 
SEND,(p) action at Process 7 is in a separate class with associated class time, say tn. 
We do so because each Process 7 is a distinct system component that controls the 


sending of its own messages. 


When we compose automata, the state set of the resulting automaton is the cross 
product of the state sets of the the component automata. More interestingly, the 
timing partition of the new automaton is the union of the timing partitions of the 
component automata. Thus composition preserves the timing guarantees of the con- 


stituent automata. 


2.1.3 Specifying Correctness in the I/O Automaton Model 


How do we specify the correctness of an automaton? We take the token passing system 
as an example. Our first attempt may be to specify correctness in terms of a set of 
legal states: a token passing system is correct if in any state there is exactly one token. 


But this allows implementations in which the token always remains at Process 1. Thus 
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we also need to specify that every process receives a token within a bounded amount 
of time. To do this, we need to describe executions of the system and to model how 


time passes. 


To model how time passes, we use the concept of a timed state and a timed action. 
Intuitively, a timed state is a pair (s,t) where s is a state of the automaton and t is a 
time; ¢t is read as the time associated with state s. Similarly, a timed action is a pair 


(a,t) where a is an action of the automaton. 


When an automaton “runs”, it generates a string representing an ezecution of the 
system the automaton models. This string is simply an alternating sequence of timed 
states and timed actions, that begins with a timed start state. An execution must 
respect the state transition rules of the automaton and the timing guarantees specified 


by each class of the automaton. Section 2.2 contains a more precise description. 


Using this definition of an execution, we can say that a token passing system S is 


correct if for every execution a of S: 


e There is exactly one token in any state of a. 


e Within a bounded time after any state of a, Process i receives a token. 


Specifying correctness using a set of legal executions is a reasonable solution that 
we will use sometimes. However, the correctness specification refers to the state of the 
implementing system. Ideally, we should treat the implementing system as a “black 


box” and describe its correctness in terms of its externally visible behavior. 


Suppose we wished our token passing system to be used by other applications. Then 
we need to specify additional actions to act as an interface to such client applications. 
To do so we add two additional actions to each Process 7. The first is an output action 
DELIVER_TOKEN; and the second is an input action RETURN_TOKEN;. This is shown 
in Figure 2.3. Intuitively, DELIVER_TOKEN; is used by Process 2 to deliver a token to 
the external client; RETURN_TOKEN; is used by the external client to return the token 
to Process 1. Suppose we now compose all process and channel automata. Next, we 
reclassify all actions of the composition as internal actions except the DELIVER_TOKEN 
and RETURN_TOKEN actions. Such a reclassification can be done formally using a 


“hide” operator. 


40 


DELIVER_TOKEN | | RETURN_TOKEN ., 
1 1 


RECEIVE | PROCESS i BEND 


eC ee aa 


Figure 2.3: Process 2 with additional interfaces to an external client 
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Figure 2.4: A modular token passing system and its external interface to clients 


The resulting automaton is shown in Figure 2.4. It essentially reflects the interface 
between the token passing system and its clients. A natural question is: can we specify 


correctness of the system solely in terms of the external interface to the clients? 


To answer this question, we first define an external behavior (or behavior for short) 
of an automaton. A behavior corresponding to an execution a is the subsequence of 
a consisting only of timed input and output actions, together with a start tame. The 
start time of the behavior is equal to the time associated with the first state of a. A 
sequence # is said to be a behavior of automaton A if @ is the behavior corresponding 
to some execution of A. Clearly any behavior of the automaton in Figure 2.4 will 
consist only of DELIVER_TOKEN and RETURN_TOKEN actions. 


Using the notion of a behavior, we can define the “external” correctness of the 
token passing system as follows. A token passing system S is said to be correct if for 


any behavior £ of S the following two properties hold in the sequence £: 


e There must be a RETURN_TOKEN; between any 
DELIVER_TOKEN; and a later DELIVER_TOKEN, for any 2,7. 
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e Suppose that in 8, a RETURN_TOKEN; occurs in bounded time after every 
DELIVER_TOKEN; for all 1. Then for any 7 and any suffix y of §, a 
DELIVER_TOKEN; will occur in bounded time after the start of y. 


The first condition is a “safety” property. It guarantees that Process 7 will not 
receive the token until all processes that have received the token before 7 have returned 
the token. The second property is a “liveness property”. It ensures that Process 7 
will get the token periodically. However, we can only guarantee this property if all 
external clients return the token in bounded time after a token is delivered to them. 
Notice a modelling trick that is being used here. While an I/O automaton A must 
allow all possible inputs, we can specify that A exhibit correct behavior only on certain 


“well-formed” inputs. 


2.2. Formal Summary of the I/O Automaton Model 


We summarize our discussion so far. In this thesis, we will use the following model 
which is a special case of the model in [MMT91]. However, our terminology is slightly 
different from that of |MMT91]. 


An automaton A consists of five components: 


e a finite set of actions actions(A) that is partitioned into three sets called the set 
of input, output, and internal actions. The union of the set of input actions and 
the set of output actions is called the set of external actions. The union of the 


set of output and internal actions is called the set of locally controlled actions. 
e A finite set of states called states(A). 
e A nonempty set start(A) C states(A) of start states. 


e A transition relation R(A) C states(A) x actions( A) x states( A) with the property 


that for every state s and input action a there is a transition (s,a,5) € R(A). 


e An equivalence relation part(A) partitioning the set of locally controlled actions 
into equivalence classes, such that for each class c in part(A) we have a positive 
real upper bound ¢,. (Intuitively, t, denotes an upper bound on the time to 


perform some action in class c.) 
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An action a is said to be enabled in state s of automaton A if there exist some 
& € states( A) such that (s,a,5) € R(A). An action ais disabled in state s of automaton 
A if it is not enabled in that state. Since one action may occur multiple times in a 
sequence, we often use the word event to denote a particular occurrence of an action 


in a sequence. 


To model the passage of time we use a time sequence. A time sequence to,t1,to,... 
is a non-decreasing sequence of non-negative real numbers; also the numbers grow 
without bound if the sequence is infinite. A timed element is a tuple (z,t) where t is a 
non-negative real and z is an element drawn from an arbitrary domain. A timed state 
for automaton A is a timed element (s,¢) where s is a state of A. A timed action for 


automaton A is a timed element (a,t) where a is an action of A. 


Let X = (zo, to), (@1,t1),... be a sequence of timed elements. We will also use 


z,;.time (which is read as the time associated with element z;) to denote t¢;. 


We say that element x; occurs within time ¢ of element 2; if 7 > 2 and 2;.time < 
z;.time +t. We will use X.start (which is read as the start time of X) to denote fo. 


Definition 2.2.1 An execution a of automaton A is an alternating sequence of timed 
states and actions of A of the form (80, to), (a1, t1), ($1, t1), (G2, t2), ($2, t2),... such that 
the following conditions hold: 


1. 89 € start and (s;, 4:41, 8:41) € R for alli > 0. 


2. The sequence can either be finite or infinite, but if finite it must end with a timed 


state. 
3. The sequence to,t1,t2,... 18 a time sequence. 


4. If any action in any class c is enabled in any state s; of a then within time 
s;.time+t, either some action in c occurs or some state s; occurs in which every 


action in c is disabled. 


Notice that the time assigned to any event a; in a (i.e., a;.time) is equal to the 
time assigned to the next state (i.e., s;.time). Notice also that we have ruled out the 
possibility of so-called “Zeno executions” in which the execution is infinite but time 


stays within some bound. 
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Definition 2.2.2 Consider an execution a of A. Let y be the subsequence of a con- 
sisting of timed external actions, and let ty be the start time of a. The behavior @ 


corresponding to a is the sequence 2 = to,y. 


Notice that the start time of a behavior is the start time of the corresponding 
execution. The behaviors of automaton A are the behaviors corresponding to the 


executions of A. 


Notice that a behavior is not the same “type” of sequence as an execution since a 
behavior consists of a start time followed by a sequence of timed actions. Formally, 
a behavior sequence 2 is a sequence to, (a1, t1),(@2,t2),... such that each a; is drawn 
from some set of actions and such that fo, t,,t2,... is a time sequence. Note that any 
behavior of an automaton is a behavior sequence. We will use {.start to denote the 
first element in @; @.start can be read as the start time of behavior sequence (7. As 


before, we will use a;.tame to denote #;. 


Consider an execution a = (69, to), (a1, t1), ($1, t1), (@2,t2),..... The untimed ex- 
ecution corresponding to @ is the sequence $9, 41, 81,@2,52,-.-.. For brevity, we will 
frequently describe execution a by the corresponding untimed execution 59, a1, S2,.... 
By our notation, the time associated with any state s; (or action a;) in a is s;.time 


(or a;.time). 


Similarly consider a behavior 3 = to, (bi, t1), (bo, t2),... The untimed behavior cor- 
responding to # is the sequence b,, b2,.... We will often describe behavior @ using the 
corresponding untimed behavior 5;, b2,.... Once again by our notation, the start time 


of @ is @.start and the time associated with any action 6; in ( is b;.time. 


Notice that the time associated with the first state of an execution, s9.time, is 
allowed to be an arbitrary non-negative real number. As we see below, this assumption 
allows a clean statement of an important lemma about stabilizing automata. In the 
timed automaton model [MMT91], each class also has an associated lower bound. In 
our model, the lower bound associated with each class is implicitly assumed to be zero. 
These two assumptions (or lack of assumptions) restrict us to modelling systems in 
which the value of time is not used by the protocol. In the protocols we describe later, 


we will use time only to model liveness guarantees and to measure time complexity. 


The following lemma is useful later. 
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Lemma 2.2.3 Consider any execution a of an automaton A. Any suffic of a that 


starts with a timed start state is also an execution of A. 


Proof: In essence, the lemma follows because we have no lower bounds on the time 


between actions. Jf 


2.2.1 Composition and Hiding 


To describe a collection of automata we will use a finite index set, say J. For example, 
an index set could be the set of vertices in a graph, or the set of edges in a graph. 
Thus we often speak of a collection {A;,i € I} of automata, where I is an index set. 


Often we are interested in composing such a collection of automata. 


Before automata can be composed they must obey certain restrictions. Clearly, 
the automata cannot share internal or output actions without violating the principle 
that each action is controlled by exactly one automaton. A collection of automata 
{A;,7 € I} is said to be compatible if the collection is finite, the output actions of all 
automata are disjoint, and the internal actions of any automaton are disjoint from the 
actions of any other automaton. Notice that this allows several automata to have a 
common input action. This can be used, for instance, to model a broadcast from one 


automaton to several other automata. 


Let I be a finite index set. The composition of a compatible collection {A;,i € I} 
is denoted by A = I;-7A;. A is the automaton formed as follows: 


e An action 7 is an input action of A if 7 is the input action of some A; in the 
collection and is not the output action of some other A, in the collection. The set 
of output actions of A is the union of the output actions of the collection. The 


set of internal actions of A is the union of the internal actions of the collection. 


e The set of states of A is the cross-product of the state sets of the automata in 
the collection. The set of initial states of A is the cross-product of the initial 


state sets of the automata in the collection. 


e Let s|i denote the projection of some state s of A onto automaton A;. Then the 


transitions of A are the triples (s,a,) such that for any i € I: 
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— If ais an action of A; then (s|, a, 8|¢) is a transition of Aj. 


— If a is not an action of A; then s|i = §|:. 


e The partition of A is the union of the partitions in the collection. Thus if an 
action a belongs to some class c of any A;, then a belongs to class cin A. The 
upper bound corresponding to class cin A is the upper bound corresponding to 


class c in A;. 


Note: Suppose we simply took the set of input actions of A to be the union of 
the set of input actions of the components. Then any action that is an input action of 
say A; and an output action of say A; will be classified both as an input and output 
action of A. It is to avoid this problem that the input actions of A are defined the way 
they are. Notice also that any action of the component automata is also an action of 


the composition. 


We return to our claim that the I/O automaton model is compositional. We would 
like to show that the behavior of a composition can be inferred from the behavior of 


its components. We translate the following two lemmas from [MMT]. 


We use (| A; to represent the projection of a behavior @ of A on to some constituent 
automaton A;. The projection is the subsequence of 9 containing actions of A;. We also 
assume that @|A; inherits the times of @ in the natural way. Thus the time associated 
with any action in (|A; is the time associated with the corresponding action in 3, and 
the start time of @|A; is the start time of /. 


The first lemma shows how we can “cut” a behavior of the composition into be- 
haviors of each of the pieces. The second lemma shows how we can “paste” a sequence 


of behaviors of the pieces into a behavior of the composition. 


Lemma 2.2.4 Cut Lemma Let {A;,i € I} be a compatible collection of automata 
and let A = IljcrA;. Let 8 be any behavior of A. Then 8|A; is a behavior of A; for 
everyi El. 


Lemma 2.2.5 Paste Lemma Let {A;,i € I} be a compatible collection of automata 
and let A = IljczA;. Let GB be a behavior sequence such that each action in GB is an 
external action of A. If B|A; is a behavior of A; for everyi € I, then B is a behavior 
of A. 
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Finally, there is a hiding operator on automaton A with respect to some subset & 
of the actions of A. The result of this operation is to to create a new automaton that 


is identical to A except that all actions in © are reclassified as internal actions. 


2.2.2. Useful Notation 


In all the automata that we will describe in this thesis, the state of an automaton 
consists of values assigned to a set of variables. We use record notation to extract the 
values of specific variables in the state. We say that a variable x has a value v in state 


s if s.2 =v. We sometimes omit the state s if it is clear by context which s we mean. 


When we refer to the state s; in execution a we mean the 1-th state in the sequence 
a. An interval of an execution a is a contiguous subsequence of a that starts and ends 
with a timed state. The duration of an interval |s;,s;] is s;.time—s;.time. An interval 


of a behavior £ is a contiguous subsequence of 8. 


A predicate of automaton A is a subset of the states of A. A predicate S is described 
by a Boolean formula on variables; a state s € S iff the values of the variables of A 
in state s satisfy the Boolean formula. Thus if S is e = 3 then s € S iff sz = 3. We 
also say that S holds in state s or S is true in state s to mean s € S. We say that a 
predicate S remains true for time ¢ after state s; in execution a if for all states s; that 


occur within time t of s; in a, 8; € S. 


2.2.3 Modelling Asynchronous Protocols 


In this thesis we will study “asynchronous” protocols. We wish such protocols to work 
regardless of timing assumptions. This is typically done using “fairness assumptions” 
instead of timing assumptions. However, we can model essentially the same thing by 


ensuring that our protocols work regardless of the value of t. assigned to any class c. 


The advantage of using parameterized upper bounds for classes comes in measuring 
time complexity. In the standard approach, after first proving correctness using “fair” 
executions, time complexity is then measured using the the assumption that the class 
times ¢, are constants like 1 or 0. Often the time complexity arguments are extremely 
similar to the liveness arguments used in the proof of correctness. In our approach 


there is no need for this double effort; we replace liveness arguments by showing time 


AT 


bounds on certain events occurring. These time bounds are parameterized in terms 
of the class times ¢,. Thus to obtain time complexity measures, we simply substitute 


particular values for t,. 
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Chapter 3 


Stabilization: Definitions and 


Properties 


In this chapter we will describe the basic definitions of stabilization we will use for 
the rest of the thesis. We begin in the first section with a state-based definition of 
stabilization that corresponds to the standard definitions in the literature. In the next 
section we describe a new definition of stabilization in terms of external behaviors. In 
Section 3.4 we describe a two stage proof technique for proving stabilization properties. 
Then in Section 3.5 we describe a modularity result. This result shows that (under 
certain conditions) we can prove facts about the stabilization of a big system by proving 


facts about the stabilization of each of its parts. 


3.1 Definitions of Stabilization based on Executions 


All the existing definitions of stabilization are in terms of the states and executions 
of a system. We will begin with a definition of stabilization that corresponds to the 
standard definitions (for example, that of Katz and Perry [KP90]). In the next section, 
we will describe another definition of stabilization in terms of external behaviors. We 
believe that the definition of behavior stabilization is appropriate for large systems 
that require modular proofs. However, the definition of execution stabilization given 
below is essential in order to prove results about behavior stabilization. We begin with 


the definition of execution stabilization since it is also the definition that most readers 


49 


are familiar with. 


Suppose we define the correctness of an automaton in terms of a set C’ of legal 
executions. For example, recall that for the token passing system of Chapter 2 we 
defined the legal executions to be those in which there is exactly one token in every 


state, and in which every process periodically receives a token. 


What do we mean when we say that an automaton A stabilizes to the executions 
in set Cin time t? Intuitively, we mean that within time f, all executions of A begin to 
“look like” an execution in set C’. For example, suppose C’ is the set of legal executions 
of a token passing system. Then in the initial state of A there may be zero or more 
tokens. However, the definition requires that within time t of the start of any execution 


of A, there is exactly one token in any state. 


To formalize this, we begin with the definition of a ¢-suffix of an execution a. 
Intuitively, this is a suffix of a whose first element occurs no more than time ¢ after 
the start of a. Although we will apply this definition only to executions, we will state 
the definition in terms of an arbitrary sequence of timed elements. Recall from Chapter 
2 that a timed element is a tuple (z,t) where z is either a state or an action, and ¢ is 


a time. 


Definition 3.1.1 Consider any two sequences of timed elements a and a’. We say 


that a’ is a t-suffiz of execution a tf: 


e a'.start — a.start <t and 


e a’ is a suffiz of a. 
We can now formally define execution stabilization to a set of executions: 


Definition 3.1.2 Let C’ be a set of sequences of timed elements. We say that automa- 
ton A stabilizes to the executions in C in time t if for every execution a of A there is 


some t-suffiz of execution a that is in C’. 


We also make the accompanying definition of execution stabilization to another 


automaton: 
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Definition 3.1.3 We say that automaton A stabilizes to the executions of automaton 
B in time t if for every execution a of A there is some t-suffiz of execution a that is 


an execution of B. 


So far, we have not made any assumptions about the automaton A that stabilizes 
to the executions specified by a set C or another automaton B. However, recall 
that in Dijkstra’s original definition, an automaton is “self-stabilizing” if regardless of 
what state it starts in, an automaton “eventually” produces “legal” executions. Our 
definitions are more general than Dijkstra’s definitions. To see this, we now extend 


Dijkstra’s notion of “self-stabilization” to our timed setting. 


As a stepping stone, for any automaton A we define U(A) (which can be read as 
the unrestricted version of A) to be the automaton that is identical to A except that 
any state of A can be a start state of U(A). 


Definition 3.1.4 For any automaton A, we let U(A) denote the automaton that is 
identical to A except that start(U(A)) = states(A). 


Next, we can say that an automaton A “self-stabilizes” in time t if the following 
holds: even if we start A in a state other than one of it’s start states, the resulting 
execution will begin to “look like” a properly initialized execution of A within time ft. 


Formally: 


Definition 3.1.5 We say that an automaton A self-stabilizes in time t if U(A) stabi- 


lizes to the executions of A in time t. 


The following simple lemma shows that execution stabilization is transitive. This 
is an important lemma because it allows to prove execution stabilization in several 


stages. 


Lemma 3.1.6 If automaton A stabilizes to the executions of automaton B in time t, 
and B stabilizes to the executions of automaton C in time to, then A stabilizes to the 


executions of C in time t, + to. 
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3.2 Definitions of Stabilization based on External 


Behavior 


In Chapter 2, we argued that a major theme of the I/O Automaton model [LT89] is 
the focus on external behaviors of an automaton. For instance, the correctness of an 
automaton is specified in terms of its external behaviors. In Chapter 2 for instance, we 
showed how to specify the correctness of a token passing system without any reference 
to the state of the system. We did this by specifying the ways in which token delivery 


and token return actions can be interleaved 


Thus it natural to look for a definition of stabilization in terms of external behav- 
iors. We would also hope that such a definition would allow us to modularly “compose” 
results about the stabilization of parts of a system to yield stabilization results about 


the whole system. 


Typically, the correctness of an IOA is specified by a set of legal behaviors P. An 
IOA A is said to solve P if the behaviors of A are contained in P. For stabilization, how- 
ever, it is reasonable to weaken this definition and ask only that an automaton exhibit 
correct behavior after some finite time. In most of this thesis, we will use the behavior 


stabilization definitions for specifying the stabilization properties of a system. 


As in the case of execution stabilization, we begin with the definition of a t-suffix 
of a behavior @. Intuitively, this is a portion of @ that starts at time no more than t 
after the start of 3. However, this is not as easy as defining a t-suffix of an execution. 
Recall that a behavior 8 = (to,7) consists of two components: a start time t) and a 
sequence of timed actions y. Thus we cannot simply define a t-suffix of 8 to be a suffix 
of @ as we did in the case of an execution. We would also like a ¢t-suffix of @ to be a 
behavior sequence: thus the t-suffix must have a start time as well as a sequence of 


timed actions. 


Figure 3.1 is a pictorial view of a behavior @. The second row represents the 
sequence of actions of the behavior and the first row represents the times corresponding 
to each action as well as the start time of the behavior. The dashed line to the right of 
the start time represents an instant of time that occurs no more than time ¢ after the 
start of the behavior. Now consider the behavior that starts at the time corresponding 
to the dashed line and consisting of all actions to the right of the dashed line. We 


will call such a behavior a t-suffix of behavior @. Intuitively, as we said before, this 
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Interval of time that is no more than t 


Start Time 
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Figure 3.1: A pictorial view of a t-suffix of a behavior 


represents a portion of @. However the portion starts at time no more than ¢ after the 


start of @. Formally: 


Definition 3.2.1 Consider any two behavior sequences 3 = to,y and B' = th, y'. We 
say that @' is a t-suffiz of behavior BG if: 


e B'.start — G.start < t. 


e 7’ is a suffiz of y containing all actions in B that occur at times strictly greater 
than 8’ .start. 


Note that the definition allows y' to contain some, none, or all of the actions in 
8 that occur at times equal to (’.start = tj. Note that the definition is similar but 
yet different from the definition of a ¢-suffix of an execution. Using this, we can now 


define behavior stabilization analogous to execution stabilization: 


Definition 3.2.2 Let P be a set of behavior sequences. An IOA A stabilizes to the 
behaviors in P in time t if for every behavior 3 of A there is a t-suffiz of behavior B 
that is in P. 


An automaton is said to solve another automaton B if every behavior of A is a 
behavior of B. Similarly, we can specify that A stabilizes to the behaviors of some 


other automaton B. 
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Definition 3.2.3 An automaton A is said to stabilize to the behaviors of another 
automaton B in time t if for every behavior GB of A there is a t-suffiz of behavior 3 
that is a behavior of B. 


The following lemmas are “obvious” in that they are what we expect to be true. 
Lemma 3.2.4 Every automaton stabilizes to its own behaviors in time 0. 


Lemma 3.2.5 If every behavior of automaton A is a behavior of automaton B, then 
A stabilizes to the behaviors of B in time 0. 


The next lemma is a transitivity result for behavior stabilization. It allow us to 


prove behavior stabilization results in stages. 


Lemma 3.2.6 If automaton A stabilizes to the behaviors of automaton B in time t, 
and B stabilizes to the behaviors of automaton C' in time tz, then A stabilizes to the 
behaviors of C in time t, + fo. 


Another obvious consequence of our definition is: 


Lemma 3.2.7 If automaton A stabilizes to the behaviors of an automaton B in time 
t andt >t then A stabilizes to the behaviors of B in time t. 


The previous lemma motivates a natural complexity measure called the stabilization 
time from A to B. Intuitively, this is the smallest time after which we are guaranteed 
that A will behave like B. However, since we are dealing with a potentially infinite 


set of behaviors we have to be a little more careful. 


Definition 3.2.8 The stabilization time from A to B is the infimum of allt such that 
A stabilizes to the behaviors of B in time t. 


The next lemma is simple but important because it ties together the execution and 
behavior stabilization definitions. It states that execution stabilization implies behav- 
ior stabilization. In fact, the only method we know to prove a behavior stabilization 
result is to first prove a corresponding execution stabilization result, and then use this 
lemma. Thus the behavior and execution stabilization definitions complement each 
other in this thesis: the former is typically used for specification and the latter is often 


used for proofs. 
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Lemma 3.2.9 If automaton A stabilizes to the executions of automaton B in time t 


then automaton A stabilizes to the behaviors B in time t. 


Proof: Let @ be any behavior of A. Let a be some execution of A corresponding to 
3. From the hypothesis, there is some a’ that is a t-suffix of execution a and is also 
an execution of B. Consider the behavior (’ of B corresponding to execution a’ of B. 
From the definitions, we can verify that that @’ is a t-suffix of behavior @. [J 


3.3 Discussion on the Stabilization Definitions 


First, notice that we have defined what it means for an arbitrary IOA to stabilize to 
some target set or automaton. For most of the thesis we will be interested in proving 
stabilization properties only for a special kind of automata: unrestricted automata. 
An unrestricted IOA (see Section 3.5) is one in which all states of the automaton are 
also start states. Such an IOA models a system that has been placed in an arbitrary 
initial state by an arbitrary initial fault. However, (this important observation is due 
to Arora and Gouda [AG92]), we might also be interested in modelling the response 
of a system to more restricted kinds of initial faults. Such restricted faults initially 
place a system A in some subset L of the states of A. After the initial fault, we 
would like A to behave like some other automaton B. Thus our general definitions of 
stabilization are applicable to other, more restricted forms of initial faults. While we 
will not mention this explicitly from this point on, many of the techniques developed 


in this thesis can also be applied to more restricted initial fault models. 


Next, it is reasonable to ask whether our definitions are sufficient to cover all cases 
that arise in practice? They do cover all the examples in this thesis. There are two 
places we can weaken our definitions. The first is that instead of requiring that all 
behaviors (or executions) of A stabilize in time ¢, we might only that certain “well- 
formed” behaviors (or executions) of A stabilize. A “well-formed” behavior (execution) 
is a behavior (execution) with certain restrictions on the input actions. For instance, if 
A can pass a token to the external environment E, we might only require stabilization 


for those behaviors (executions) in which F returns the token to A in bounded time. 


The second possible modification (which is sometimes needed, though again not 
in this thesis) is to only require (in Definitions 3.2.2 and 3.1.2) that the t-suffix of 
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a behavior (execution) be a suffiz of a behavior (execution) of the target set. One 
problem with this modified definition is that we know of no good proof technique to 
prove that the behaviors (executions) of an automaton are suffizes of a specified set of 
behaviors (executions).'By contrast, it is much easier to prove that every behavior of 


an automaton has a suffix that is in a specified set. 


Thus we prefer to use the simpler definitions for this thesis. 


3.4 Proof Technique 


We begin by defining an extremely useful piece of notation that is used extensively in 
this thesis. This notation allows us to specify an IOA that is identical to another IOA 
except for start states. This is roughly the inverse of the U(A) notation that creates 


an unrestricted version of automaton A without start states. 


Definition 3.4.1 For any automaton A and any subset L of states( A), we denote by 
A|L the automaton that is identical to A except that start(A|L) = L. 


There has been a great deal of work in designing ordinary automata that have 
specific start states to solve specific problems. It would be nice to gain leverage from 
this existing body of work. Suppose we are given an IOA A that solves a set of 
behaviors P and we now wish to design an IOA B that stabilizes to the behaviors in 


P. Our goals are: 


e We would like to use as much of the design of A as possible to design B. 


e We would like to use as much of the proof that A solves P as possible to prove 
that B stabilizes to the behaviors in P. 


We now describe one way in which these goals can be achieved. The following 


lemma is immediate from the definitions. 


1Such proofs seem to involve arguments about reachable states. Familiar inductive proof techniques 


(such as invariant arguments, progress metrics etc.) do not seem to suffice for this purpose. 
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Lemma 3.4.2 Consider an automaton A, L C states(A), and a problem P. Suppose 


the following conditions are true: 


e A stabilizes to the behaviors of A\L in time t. 


e Any behavior of A|L is in P. 


Then A stabilizes to the behaviors in P in time t. 


In the next two subsections, we describe the techniques used in this thesis for 


proving the two items on the list. 


3.4.1 Proving that an Automaton Solves a Problem 


To prove that any behavior of some automaton A is a behavior contained in some 
problem P, it suffices to prove that every behavior of A is a behavior of some other 


automaton B, and that every behavior of B is in P. 


There are well-known techniques (e.g., [LT89]) to show any behavior of an au- 
tomaton A is a behavior of another automaton B. A commonly used technique is a 
refinement mapping. The basic idea is to establish a suitable mapping f between a 
state of A and a state of B. Given an execution a of A, we use f to construct a 


mapped execution of B that has the same external behavior as a. 


Theorem 3.4.3 Refinement Mapping: Let A and B be automata with the same 
set of external actions. Let t. be the upper bound on the time to perform actions in 
any class c of B. Let f be a mapping from the states of A to the states of B such that: 


1. For any start state s of A, f(s) is a start state of B. 
2. For all transitions (s,7,8) of A, either 


e x is an action of B and (f(s),7,f(8)) is a transition of B OR 
e x is not an action of B and f(s) = f(8). 


3. For any class c of B and any execution a of A, suppose some action in c is 


enabled in f(s). Then within t. time of s in a either: 
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e Some action in class c occurs OR 


e Some state § occurs such that no action in class c is enabled in f(8). 


Then every behavior of A ts a behavior of B 


Proof: Consider any execution a of A. We extend the function f to map executions 


as well as states in the following way. Let f(a) be the sequence formed from a by: 


e Removing every timed action in a that is not a timed action of B and the timed 


state following such an action. 


e Replacing every timed state (s,¢) in the remaining sequence by (f(s), t). 


Since we retain every action of B, the behavior corresponding to f(a) is the be- 


havior corresponding to a. 


Next, we verify that f(a) is an execution of B. To do so we check the four 
conditions in Definition 2.2.1. Let a = (80,to), (a1, #1), (81,t1)..... Clearly, by con- 


struction, f(a) is an alternating sequence of timed states and actions of B. Let 
F(@) = (805 to (a1 #1 (81541). ++ 


We check the four conditions in turn: 


1. First 85 = f(so) € start(B) by hypothesis. Next consider (sj, a;,,,8;,,) for all 
a> 0. Let a, be the 71+ 1’st action of B in a. Intuitively, a, is the action in a 


that generated aj,, in f(a). Clearly, a, = aj,,. 


Next, consider the smallest 7 < k such that all actions between s; and s,_1 in a 
are not actions of B. Then by construction, f(s;) = s; and f(s,) = sj,,. Also 
by the hypothesis, f(s;) = f(sz-1) and (f(sz_1), ax, f(s)) is a transition of B. 


Thus (s;,@;,,,5;,,) is a transition of B. 
2. The second condition follows trivially from the construction. 


3. The third condition follows because any subsequence of a time sequence is a time 


sequence. 
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4. Suppose some action in some class c of B is enabled in some state s; of f(a). 
Then, by construction there is a corresponding state s; in a such that if y is the 
suffix of a starting with s;, then f(7) is the suffix of f(a) starting with si. Then, 
by hypothesis, either: 


e Some action in class c of B occurs within t, time of s; in a. But by 


construction, the same action occurs within t, time of s; in f(a). 


e Some state s, occurs within ¢, time of s; in a and such that all actions in 
class c are disabled in f(s,). Let k be the smallest index such that this is 
true for sz. Now if a, is not an action of B, then since f(s,_1) = f(s) this 
contradicts the assumption that k is the smallest index with this property. 
Thus a, is an action of B and hence, by construction, (f(s, ), s,.time) occurs 
in f(y). Thus there is a state that occurs within ¢, time of s; in f(a) and 


such that no action of ¢ is enabled in this state. 


3.4.2 Proving that an Automaton Stabilizes to another Au- 


tomaton 


We give one such technique in the following definition and theorem. The technique 
is similar to techniques used for proving liveness properties (e.g., [OL82, MP91]) of 
concurrent programs. Our theorem is a small generalization of a theorem for proving 


stabilization properties that was previously proposed in [GM90]. 


Let L be a closed predicate of automaton A — once L is true in an any execution 
of A, LZ remains true for the rest of the execution. We would like to prove that in 
any execution of A, ZL becomes (and stays) true in bounded time. This implies that A 
stabilizes to the executions of A|Z in bounded time. We will describe a proof rule for 
this purpose. Intuitively, instead of proving directly that the goal LZ eventually holds 
we prove that a number of subgoals L; (each of which is a predicate of A) become and 


stay true. The L; are chosen so that L is true if all the individual L; are true. 


Each subgoal £; can be intuitively thought of as “depending” on some other set of 
subgoals. This dependence relation is formalized by a partial order <. If 7 < 1, then 
(intuitively) L; “depends” on L;. Suppose we could show that if all the subgoals that 
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L; depend on become and stay true, then L; becomes and stays true. Then we can 
concude that eventually all subgoals (and hence L) become and stay true. Intuitively, 
this follows because the dependency relation is acyclic since it is formalized using a 


partial order. 


We can extend this idea to a timed setting to show that L will become true in time 
proportional to the maximum length of a “dependency chain”. A dependency chain is 
formalized using the standard concept of a chain in a partially ordered set. Consider 
a set {L;,i1 € I} where (I,<) is a finite partially ordered set. A chain is simply a 
sequence L, < Ing <...< Lj. 


The preceding discussion motivates the following definition and theorem. 


Definition 3.4.4 We say that an automaton A is stabilized to predicate L using pred- 


icate set L and time constant t if: 


1. £L= {L;,i € I} of sets of states of A, where (I,<) is a finite, partially ordered 
index set. We let height(L) denote the mazimum length chain in the partial 


order. 
2. Mier Lh; CL. 


3. For alli € I and for all steps (s,7,8) of A, if s belongs to Nj<;L;, then & belongs 
to L;. 


4. For everyit € I and every execution a of A and every state s ina the following 
is true. Suppose that either s € Nj<:L; or there is no L; < L;. Then there 1s 


some state 6 € L; that occurs within time t of s ina. 


The first condition says there is a partial order on the predicates in £. The second 
says that L becomes “true”, when all the predicates in £ become true. The third is 
a stability condition. It says that any transition of A leaves a predicate DL; true, if all 
the predicates less than or equal to L; are true in the previous state. Finally the last 
item is a liveness condition. It says that if all all the predicates strictly less than L; 


are true in a state, then within time ¢ after this state, LZ; will become true. 


We define height(L;), the height of a predicate L; € £, to be the maximum length 
of a chain that ends with L; in the partial order. The value of height(L) is, of course, 
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the maximum height of any predicate L; € £. By the liveness condition, within time 
t all predicates with height 1 become true; these predicates stay true for the rest of 
the execution because of the third stability condition. In general, we can prove by 
induction that within time z-t all predicates with height 2 become and stay true. This 


leads to a simple but useful theorem: 


Theorem 3.4.5 Execution Convergence: Suppose that automaton A is stabilized 
to predicate L using predicate set L and time constant t. Then, A stabilizes to the 
executions of A|L in time height(L) - t. 


Proof: By induction on h,0 < h < height(L), in the following inductive hypothesis. 


Inductive Hypothesis: There is some state s that occurs within time h-t of the 
start of a such that s € L; for all L; € £ with height(L;) < h. 


The inductive hypothesis implies that there is some state s that occurs within time 
h- height(L) of the start of a and such that s © L. The theorem follows from this fact 
taken together with Lemma 2.2.3. 


Base Case, h = 0: Follows trivially since there is no L; € £ with height(L;) = 0. 
Inductive Step, 0 < h < height(£) — 1: Assume it is true for h. Then there is 


some state s; that occurs within time h-t of the start of a and such that for all L; € £ 
with height(L;) < h, s; € L;. Consider any L; € £ with height(L;) = h+1. By the 
fourth condition in Definition 3.4.4, we know there must some state s4(;) € L; that 
occurs within time t of s; in a. Let k = Maz{f(j) : height(L;) =h+1}. Then from 
the third condition in Definition 3.4.4, we see that s, occurs within time (h + 1) -t of 
the start of a and such that for all L; € £ with height(Z;)<h+1,s,EL; IJ 


3.5 Modularity Theorem 


We will mostly deal with stabilization properties of a special class of automata called 
unrestricted automata or UIOA. Intuitively, a UIOA models a system that can start 


in an arbitrary state. 


Definition 3.5.1 A UIOA A is an automaton such that start(A) = states(A) (i.e., 


all states are start states). 
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However, we will often show that a UIOA A stabilizes to the behaviors of a second 
special kind of automaton called a Closed I/O Automaton or a CIOA . We define the 


reachable states of an automaton A to be the states that can occur in executions of A. 


Definition 3.5.2 A CIOA is an automaton such that every reachable state is also a 


start state. 


It is easy to see that: 


Lemma 3.5.3 Every UIOA 1s a CIOA . 


The two following lemmas are extremely convenient and are used often below with- 
out explicit reference. It is the reason we allow executions and behaviors to start with 
arbitrary values of time. Also, the next two lemmas depend crucially on the fact that 


there are no lower bounds on the time between actions. 


Lemma 3.5.4 Consider any execution a of a CIOA A. Any suffix of a that starts 


with a timed state is also an execution of A. 


Proof: Follows directly from Lemma 2.2.3 and the definition of a CIOA. J 


Suppose we begin to view an automaton after it has “run for a while” and the 
resulting behavior is indistinguishable from an ordinary behavior of the automaton. 


Then, intuitively, we say that the automaton is suffiz-closed. More formally: 


Definition 3.5.5 We say that an automaton A is suffiz-closed if for every behavior 
B of A and every time t > 0, every t-suffiz of behavior B is a behavior of A. 


A remarkable number of interesting automata we will study in this thesis are suffix- 


closed. This fact is explained by the following lemma: 


Lemma 3.5.6 Any CIOA A is suffiz-closed. 
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Suffix of behavior includes all 
actions after this point in time. 


Behavior a a Bae: VA a 
Execution So eee Ye: ee: ee er enne* | 


Figure 3.2: Obtaining the suffix of an execution corresponding to a t-suffix of the behavior of the execution. 


Proof: We will only sketch the main idea of the proof. Consider any behavior £ of A 
and any t > 0. Let §’ be any t-suffix of behavior @. Consider any execution a of A 
such that the behavior of a is 8. The proof consists of using a to construct another 
execution a’ of A such that the behavior of a’ is b’; a’ is essentially a suffix of a whose 


start time is adjusted to match the start time of (’. 


The behavior 8 and corresponding execution a@ are sketched in Figure 3.2. By 
definition, for every action in @ there is a corresponding external action in a which 
occurs at the same time. This is sketched by drawing the action in the behavior directly 
above the corresponding action in the execution. (However, since the execution will, in 
general, have internal actions not included in the behavior, the indices of the actions 


will not necessarily match. Thus in the figure a, in @ corresponds to a, in a.) 


The t-suffix @’ can be sketched using a line that contains all actions in @ occuring 
to the right of the line (see Figure 3.2). The line is drawn between two actions in @ 


because the start time of 8’ may occur in between the times of two actions in (. 


We need a suffix a’ of a whose behavior is equal to 3’. Thus we look for a state 
Sz in @ corresponding to the vertical time line drawn in Figure 3.2. But we may not 
have a state in a whose time is equal to the start time of @’. So (intuitively) we choose 
s, to be the first state that occurs to the “left” of the vertical time line. Then we 
choose a’ to be the suffix of a starting with s, and with the time of s, adjusted to 
be equal to @’.start. This works for two reasons. First, by definition of a CIOA, s, 
is a start state of A. Second, we have no lower bounds on the time between actions 
in A. Thus increasing the time of the initial state of an execution (and such that the 


resulting time is no greater than the time of the first action) still leaves us with a legal 
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execution. Jf 


The suffix-closed property is not just a interesting curiousity. It also provides the 


basis for the following important Modularity Theorem that we discuss next. 


Our Modularity Theorem about the stabilization of composed automata may seem 
“obvious”. We would expect that if each piece A; of a composed system stabilizes to the 
behaviors of say B;, then the composition of the A; should stabilize to the composition 
of the B;. Sadly, this is not quite true. There is a counterexample described in Section 
3.5.1 which shows that if we allow some of the B; to be arbitrary automata, then this 


statement is false. 


The main problem is that for a given behavior of the system A, the component 
automata A; may stabilize at different times. But if each of the A; begin to “look 
like” the corresponding B; at different times, then it may not be possible to paste the 
resulting behavior into a behavior that “looks like” a behavior of B. However, this 
problem does not arise if each of the B; is suffix-closed. Thus we have the following 


result. 


Theorem 3.5.7 Modularity: Let I be a finite index set. Let Ay = {A;,i € I} be a 
compatible set of automata and By; = {B;,i € I} be a second set of compatible, suffiz- 
closed automata. Suppose also that for alli C I, A; stabilizes to the behaviors of B; in 
time t. Let A = IljezyAz and B = IljceyBy. Then A stabilizes to the behaviors of B in 


time t. 


Proof: The proof relies on the Cut Lemma (Lemma 2.2.4) that allows us to dis- 
sect a behavior of a system into its component behaviors, and the Paste Lemma 


(Lemma 2.2.5) that allows us to paste component behaviors into a system behavior. 


Consider any behavior 8 of A. Consider the (’ that is a t-suffix of behavior @ and 
such that: 


e @' start =t+ @.start. 


e Any actions in (’ occur at times strictly greater than ¢. 


We can verify that such a @’ exists from the definition of a ¢-suffix of a behav- 
ior. Intuitively, @’ is chosen so that all component behaviors are guaranteed to have 
stabilized in 0’. 
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Now consider any i € J. By the Cut Lemma (Lemma 2.2.4), G|A; is a behavior 
of A;. But because A; stabilizes to the behaviors of B; in time t, there must be some 
t' << ¢ and some @; such that: 


e @; is a t'-suffix of @|A; and is also a behavior of B;. 


e G;.start = t' + B.start. 


Next consider §'|A;. It can be verified that 6’|A; is a t’-suffix of G@; for some t”. 
Thus by the fact that B; is suffix-closed, 6’|A; is a behavior of B;. Thus (’|A; is a 
behavior of B; for all 1. Hence by the Paste Lemma (Lemma 2.2.5), 8’ is a behavior 
of B. The theorem follows since ’ is a t-suffix of behavior @. JJ 


3.5.1 Discussion of the Modularity Theorem 


In the hypothesis of the modularity theorem, we assumed that each of the B; was suffix- 
closed. We show a counterexample to show that if the B; are allowed to be arbitrary 
automata, then the theorem is false. Consider the specification of automaton B; shown 
in Figure 3.3. Let A; be a UIOA which is identical to B; except that the start states of 
A; are unrestricted (i.e., the initial value of count; in A can be any value in the range 
{0,...,2}). 

It is easy to see that A; stabilizes to the behaviors of B; in time 3¢ because within 
that time the value of count; must reach 0. After such a state, any behavior of A; 
is a behavior of B;. Now consider an index set J = {1,2}. Consider A which is the 
composition of A, and Az and B which is the composition of B, and B,. We claim 


that A does not stabilize to B in time 3¢ (or in fact in any finite time). 


To see this, we start with the following observation. In any behavior of B in which 
the actions of B, and Bz strictly alternate, the counter values output in such a behavior 
will be of the form 0,0,1,1,2,2,0,0,.... Now consider the behavior corresponding to 
an execution of A in which count, = 0 initially and count, = 2 initially and the actions 
of A, and Ap, strictly alternate starting with A,. Then the counter values output in 
such a execution will be of the form 0,2,1,0,2,1,0,2.... From the earlier observation, 
it follows that there is no suffix of this behavior of A that is a behavior of B. 
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The state of B; consists of an integer variable count; € {0,...,2} 

The initial value of count; = 0 (* i.e., B is not a UIOA *) 

INCREMENT,(k) (*output action, outputs counter value using parameter k*) 
Precondition: k = count; 


Effect: count; := (count; + 1) mod 3 


Any INCREMENT; action is in a separate class with upper bound f. 


Figure 3.3: Specification for Automaton B; 


Suppose we weakened the definition of stabilization to allow any t-suffix of A to bea 
suffix of a behavior of B. With this weaker definition, the counterexample disappears.” 
Thus the suffix-closed requirement may be a consequence of our (stronger) stabilization 
definition. However, we did not find a way to prove the modularity theorem using 
the weaker definition. Without the suffix-closed requirement it is difficult to “paste” 
together behavior suffixes of the B;’s to create a behavior of B. A possible research 
direction would be to look for weaker conditions (than the B; being suffix-closed) 
under which the modularity theorem would work. Another research direction would 
be to extend these results to systems with non-zero lower bounds on the time between 


actions. 


3.6 Summary 


The two main contributions of this chapter are the definitions of stabilization in terms 
of external behaviors (Definitions 3.2.2 and 3.2.3) and the modularity theorem (The- 
orem 3.5.7). 


The definitions of stabilization in terms of external behaviors are different from 


2T am grateful to Robert Gallager and Victor Luchanko of MIT for pointing this out. 
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previous definitions that are in terms of the states and executions of the underlying 
automaton. The external behavior definition allow us to define that automaton A 
stabilizes to another automaton B even though A and B have different state sets. 
This is most useful when A is a low level model (e.g., an implementation) and B is a 


high level model (e.g., a specification). 


The modularity theorem gives us a formal basis for a building-block approach: it 
allows us to prove facts about the stabilization of a big system by proving facts about 
the stabilization of each of its parts. The requirement that each of the target automata 
be suffix-closed may seem restrictive. However, in a stabilizing setting this is not the 
case. Most interesting specifications (for problems such as message delivery, routing, 


and scheduling) are either suffix-closed or can be rephrased so they are suffix-closed. 


We have already seen that any UIOA (an automaton for whom every state is a start 
state) is suffix-closed. Similarly, any CIOA (an automaton for whom every reachable 
state is a start state) is suffix-closed. In a stabilizing setting the basic building blocks 
are UIOAs since they model systems that can start in an arbitrary state. The methods 
developed in this thesis, on the other hand, tend to construct automata that are 
CIOAs.? Thus we can build stabilizing solutions modularly in several stages. In the 
first stage we compose s set of UIOAs to yield a CIOA. In the next stage, the resulting 
CIOAs are composed with other UIOAs to yield more CIOAs. This process can be 
repeated indefinitely to build a complex stabilizing solution to a problem. Since all the 
pieces used are suffix-closed, the modularity theorem can be used at each stage. As 
an example, the stabilizing spanning tree protocol of Chapter 8 is constructed using 
the stabilizing reset protocol of Chapter 7 which in turn can be constructed using a 
stabilizing Data Link implementation. Thus, despite its restrictions, the modularity 


theorem is extremely useful in this thesis. 


We also have a definition of stabilization in terms of executions that corresponds 
to the standard definition ( Definition 3.1.2). This definition in terms of executions is 
used in this thesis for two reasons. First, it is sometimes useful in its own right when 


the alternative would entail adding many superfluous actions.* Second, the definition 


3Our methods construct automata that stabilize to a specification automaton of the form A|L. If L 
is a closed predicate of A —i,e., no transition of A can falsify Z - then A|L is a CIOA. 

4For example, the correctness of a spanning tree protocol is easily defined in terms of a parent 
variable at each node. For correctness, we could specify that the graph induced by the parent variables 


be a spanning tree of the network. An external behavior specification would require additional output 
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in terms of executions is essential for proving results about stabilization of behaviors 


because our main tool for proving such results is Lemma 3.2.9. 


The definitions we use give us several nice properties that we believe any definition 
of stabilization should provide. The properties we believe to be important are: transi- 
tivity for both behavior and execution stabilization (Lemma 3.2.6 and Lemma 3.1.6), 
the fact that execution stabilization implies behavior stabilization (Lemma 3.2.9), the 


fact that stabilization in time t implies stabilization in time greater than t (Lemma 


3.2.7), and the Modularity theorem (Theorem 3.5.7). 


The index contains pointers to the definitions given in the last two chapters. The 


appendix also contains a list of commonly used notation for easy reference. 


actions to report the value of the parent variables at each node. 
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Chapter 4 


Local Checking and Correction in a 


Shared Memory Model 


In Dijkstra’s [Dij74] model, a network protocol is modelled using a graph of finite 
state machines. In a single move, a single node is allowed to read the state of its 
neighbors, compute, and then possibly change its state. In a real distributed system 
such atomic communication is impossible. Typically communication has to proceed 
through channels. Such channels must be be modelled explicitly as state machines that 
can store messages sent from one node to another. Also, in message passing models, the 
channel state machine is essentially fixed (with actions to send and deliver packets) but 
the node state machines can be arbitrarily specified by the protocol designer. However, 
in Dijkstra’s model all state machines are node state machines and can be arbitrarily 


specified by the protocol designer. 


While Dijkstra’s original model is not very realistic, it is probably the simplest 
model of an asynchronous distributed system. This simple model provided an ideal 
vehicle for introducing |Dij74] the concept of stabilization without undue complexity. 
For this chapter only, we will use Dijkstra’s original model to introduce the method of 
local checking and correction. In later chapters, we will use a more realistic message 


passing model. Thus the goals of this chapter are: 


e To describe some simple examples of local checking and correction that are more 


interesting than than the trivial examples given in Chapter 1. 
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e To show that existing work in [Dij74] and [AG90] can be understood very suc- 


cinctly using the framework of local checking and correction. 


The main result of the chapter is a theorem (Theorem 4.3.1) that states that 
any locally checkable protocol on a tree can be efficiently stabilized. To motivate this 
theorem, we begin in Section 4.2 with a reset protocol [AG90] due to Arora and Gouda. 
We examine the behavior of the Arora-Gouda protocol in a good state and conclude 
that the protocol is in a good state when all link subsystems are in a good state. Then 
we show how to add correction actions to the protocol such that if a link subsystem 
is in a bad state, it can be corrected to a good state. We also determine an order 
in which link subsystems can be corrected so as to ensure that the correction process 


converges. 


In Section 4.3 we generalize the procedure followed in Section 4.2 to obtain The- 
orem 4.3.1. Then in Section 4.4 we show how one of Dijkstra’s protocols [Dij74] can 


easily be understood using Theorem 4.3.1. 


4.1 Modelling Shared Memory Protocols 


We will use the version of the timed I/O Automaton model [MMT91] described in 
Chapter 2. How can we map Dijkstra’s model into this model? Suppose each node in 
Dijkstra’s model is a separate automaton. Then in the Input/Output automata model, 
it is not possible to model the simultaneous reading of the state of neighboring nodes. 
The solution we use is to dispense with modularity and model the entire network as a 
single automaton. All actions, such as reading the state of neighbors and computing, 
are internal actions. The asynchrony in the system, which Dijkstra modelled using a 
“demon”, is naturally a part of our model. Also, we will describe the correctness of 


Dijkstra’s systems in terms of executions of the automaton. 


4.2 A Reset Protocol on a Tree 


Before describing the reset protocol due to Arora and Gouda [AG90], we first describe 


the network reset problem. 
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Recall that we have a collection of nodes that communicate by reading the state 
of their neighbors. The interconnection topology is described by an arbitrary graph. 
Assume that we are given some application protocol that is being executed by the 
nodes. We wish to superimpose a reset protocol over this application such that when 
the reset protocol is executed the application protocol is “reset” to some “legal” global 
state. A “legal” global state is allowed to be any global state that is reachable by the 
application protocol after correct initialization. The problem is called distributed reset 


because reset requests may arrive at any node. 


A simple and elegant network reset protocol is due to Finn [Fin79]. In this protocol 
each node 1 running the application protocol has a session number. When the reset 
protocol is not running, the session numbers at every node are the same. When a node 
receives a reset request, it resets the local state of the application (to some prespecified 
initial state) and increments its session number by 1. When a node sees that a neighbor 
has a higher session number, it changes its session number to the higher number and 
resets the application. Finally, the application protocol is modified so that a node 
cannot make a move until its session number is the same as that of its neighbors. This 
check prevents older instances of the application protocol from “communicating” with 
newer instances of the protocol. This protocol is shown to be correct [Fin79] if all the 


session numbers are initially zero and the session numbers are allowed to grow without 


bound. 


We rule out the use of unbounded session numbers as unrealistic. Also, in a stabi- 
lizing setting, having a “large enough” size for a session number does not work. This 
is because the reset protocol can be initialized with all session numbers at their max- 
imum value. Thus, we are motivated to search for a reset protocol that uses bounded 
session numbers. Suppose we could design a a reset protocol with unbounded numbers 
in which the difference between the session numbers at any two nodes is at most one in 
any state. Suppose also that for any pair of neighboring nodes u and v that compare 
session numbers, the session number of one of the nodes (say w) is always no less than 
the session number of the other node. Then, since the session numbers are only used 
for comparisons, it suffices to replace the session numbers by a single bit that we call 
sbit;. This is the first idea in Arora and Gouda’s reset protocol [AG90]. 


To realize this idea, we cannot allow a node to increment its session number as 
soon as it gets a reset request. Otherwise, multiple reset requests at the same node 


will cause the difference in session numbers to grow without bound. Thus nodes must 
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coordinate before they increment session numbers. 


In Arora and Gouda’s reset protocol |AG90], the coordination is done over a rooted 
tree. Arora and Gouda first show how to build a rooted tree in a stabilizing fashion. 
In what follows we will assume that the tree has already been built. Thus every node 
i has a pointer called parent(i) that points to its parent in the tree and the parent of 


the root is a special value nil. 


Given a tree, an immediate idea is to funnel all reset requests to the root. On receipt 
of a request, the root could send reset grants down the tree. Nodes could increment 
their session number on receiving a grant. Unfortunately, this does not work either 
because a node A in the tree may send a reset request and receive a grant before some 
other node B in the tree receives a grant. After getting its first grant, 4 may send 
another request and receive a second grant before B gets its first grant. Assuming 
that the session numbers are unbounded, the difference in the session numbers of A 


and B can grow without bound. 


Instead, the reset task is broken into three phases. In the first phase, a node sends 
a reset request up the tree towards the root. In the second phase, the root sends a 
reset wave down the tree. In the third phase, the root waits until the reset wave has 
reached every node in the tree before starting a new reset phase. This ensures that 
after the system stabilizes,’ the use of three phases will guarantee that a single bit 


sbit; is sufficient to distinguish instances of the application protocol. 


The three phases are implemented by a mode variable mode; at each node 7. The 
mode at node i has one of three possible values: init, reset, and normal. All nodes are 
in the normal mode when no reset is in progress. To initiate a reset, a node z sets 
mode; to init (this can be done only if both 7 and its parent are in normal mode). A 
reset request is propagated upwards by the action PROPAGATE_REQUEST;, which sets 
the mode of the parent to init when the mode of the child is znit. A reset wave is begun 
by the root by the action START_RESET which sets the mode of the root to reset. The 
reset wave propagates downwards by PROPAGATE_RESET; which sets the mode of a 
child to reset if the mode of the parent is reset. When a node changes its mode to 
reset, it flips its session number bit, and resets the application protocol. Finally, the 
completion wave is propagated by the action PROPAGATE_COMPLETION; which sets a 


node’s mode to normal when all the node’s children have normal mode. 


lin this chapter, we will always use the execution stabilization definitions 
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The automaton code for this implementation is shown in Figure 4.1 and Figure 4.2. 
Notice that besides the actions we have already described, there is a CORRECT; action 
in Figure 4.2. This action was used in an earlier version [AG90| to ensure that the 


reset protocol was stabilizing. 


Informally, the reset protocol is stabilizing if after bounded time, any reset requests 
will cause the application protocol to be properly reset. The correction action in 
Figure 4.2 [AG90] ensures stabilization in a very ingenious way. However, the proof of 
stabilization is somewhat difficult and not as intuitive as one might like. The reader 
is referred to |[AG90] for details. Instead, we will use local checking and correction to 
describe another correction procedure that is very intuitive. As a result, the proof of 


stabilization becomes transparent. 


We start by writing down the “good” states of the reset system in terms of link 
predicates L;;. We say that the system is in a good state if for all neighboring nodes 


a and j, the predicate L;,; holds, where L;,; is the conjunction of the two predicates: 


e If (parent(t) = 7) and (mode; # reset) then (mode; # reset) and (sbit; = sbit;) 
e If (parent(t) = 7) and (mode; = reset) then either: 


— (mode; # reset) and (sbit; ~ sbit;) OR 
_ (sbit; = sbit;) 


The predicates can be understood intuitively as describing states that occur when 
the reset system is working correctly. The first predicate says that if the parent’s mode 
is not reset, then the child’s mode is not reset and the two session bits are the same. 
This is true when the system is working correctly because of two reasons. First, the 
child enters reset mode only when its parent is in that mode, and the parent does not 
leave reset mode until the child has left reset mode. Second, if the parent changes its 
session bit, the parent also goes into reset mode; and the child only changes its session 


bit when the parent’s mode is reset. 


The second predicate describes the correct states during the second and third 
phases of the reset until the instant that the completion wave reaches 7. It says 
that if the parent’s mode is reset, then there are two possibilities. If the child has 
not “noticed” that the parent’s state is reset, then the child’s bit is not equal to the 
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The state of the system consists of two variables for every process in the tree: 
mode; € {init, normal, reset} 


sbit;, a bit 


PROPAGATE_REQUEST; (*internal action to propagate a reset request upwards *) 
Preconditions: 
mode; = normal 
i = parent(j) and mode; = init 


Effects: mode; := init 


START_RESET; (*internal action at root to start a reset wave*) 
Preconditions: mode; = init and parent(i) = nil 
Effects: 
mode; := reset; (*also reset application state at this node*) 


sbit; :=~ sbit;; (“flip bit*) 


PROPAGATE_RESET; (*internal action to propagate reset downwards”) 
Preconditions: 
mode; # reset 
j = parent(i) and mode; = reset and sbit; # sbit; 
Effects: 
mode; := reset; (*also reset application state at this node*) 


sbit; :=~ sbit;; (*flip bit*) 


PROPAGATE_COMPLETION; (*internal action to propagate completion wave upwards *) 
Preconditions: 
mode; = reset 
For all children j of i: mode; = normal and sbit; = sbit; 


Effects: mode; := normal 


Every action is in a separate class with upper bound equal to 
tn 


Figure 4.1: Normal Actions at node z in Arora and Gouda’s Reset Protocol.[AG90] 
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CorreEcT; (*extra internal action for correction at node 7*) 
Preconditions: 
j = parent(i) # nil 
(mode; = mode;) and (sbit; # sbit;) 
Effects: sbit; := sbit; 


Every action is in a separate class with upper bound equal to 
tn 


Figure 4.2: Original correction action in Arora and Gouda’s Reset Protocol [AG90]. 


parent’s bit. (This follows because when the parent changes its mode to reset, the 
parent also changes its bit; and just before such an action the second predicate assures 
us that the two bits are the same.) On the other hand, if the child has noticed that the 
parent’s state is reset, then the two bits are the same. (This follows because when the 
child notices that the parent’s mode is reset, the child sets its bit equal to the parent’s 


bit and does not change its bit until the parent changes its mode.) 


Suppose that in some state s these link predicates hold for all links in the tree. 
Then [AG90] show that the system will execute reset requests correctly in any state 
starting with s. This is not very hard to believe. But it means that all we have to 
do is to add correction actions so that all link predicates will become true in bounded 


time. 


The tree topology once again suggests a simple strategy. We remove the old ac- 
tion CORRECT; in Figure 4.2 and add a new action CORRECT_CHILD; as shown in 
Figure 4.3. Basically, CORRECT_CHILD; checks whether the link predicate on the link 
between 2 and its parent is true. If not, 2 changes its state such that L,; becomes true. 
Notice that CORRECT_CHILD; leaves the state of 2’s parent unchanged. Suppose 7 is 
the parent of 2 and k is the parent of 7. Then CORRECT_CHILD, will leave L;, true if 


L;, was true in the previous state. 


Thus we have an important stability property: correcting a link does not affect the 
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CorRRECT_CHILD; (*modified correction action at nodes”) 
Preconditions: 
j = parent(i) # nil 
L;,; does not hold 
Effects: 
sbit; := sbit; 


mode; := mode; 


All actions are in a separate class with upper bound ¢,. 


Figure 4.3: Modified Correction action for Arora and Gouda’s Reset Protocol. All other actions are as in 
Figure 4.1. 


correctness of links above it in the tree. Using this we can show that in bounded time, 
all links will be in a good state and so the system is in a good state. Rather than 
prove that this modified automaton stabilizes, we will prove a more general result in 
the next section: that any locally checkable tree automaton can be locally corrected 


into a good global state. 


We will return to the network reset problem in Chapter 7. Our stabilizing reset 
protocol is more efficient than the reset protocol of [AG90] and is also designed to 


work in a message passing model. 


4.3 Tree Correction for Shared Memory Systems 


In the last section, we described informally the problem of stabilizing a reset protocol 
on a tree. We also suggested a technique of adding correction actions to every node. 
That example motivates us to ask whether there is a general result for trees. To 


describe and prove such a general result, we start with the following definitions. 


We will continue to model a network as a single automaton in which a node can 
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read and write the state of its neighbors in a single move using an internal action. 


Formally: 


A shared memory network automaton N for graph G = (E£,V) is an automaton in 


which: 


e The state of NV is the cross-product of a set of node states, S,,(M), one for each 
node u € V. For any state s of N, we use s|u to denote s projected onto S,. 


This is also read as the state of node wu in global state s. 


e All actions of NV are internal actions and are partitioned into sets, A,(NV), one 
for each node u € V 


e Suppose (s,7, 5) is a transition of NV and z belongs to A,(N). Consider any state 
s' of N such that s’|u = s|u and s’|v = s|v for all neighbors v of u. Then there 
is some transition (s’,7, 5’) of NV such that 3’\v = 3|v for u and all u’s neighbors 


in G. 


e Suppose (s,7,5) is a transition of NV and 7 belongs to A,(N). Then s|v = 3\v 
for allv #u. 


Informally, the third condition requires that the transitions of a node u € V only 
depend on the state of node u and the states of of the neighbors of u in G. The fourth 
condition requires that the effect of a transition assigned to node u € V can only be 


to change the state of u. 


A shared memory tree automaton is a shared memory network automaton where G 
is a rooted tree. Thus for any node i in a tree automaton, we assume there is a value 
parent(z) that points to the parent of node i in the tree. There is also a unique root 
node r that has parent(r) = nil. For our purposes, it is convenient to model the parent 
values as being part of the code at each node. More generally, the parent pointers 
could be variables that are set by a stabilizing spanning tree protocol as shown in 


[AG9O]. 


In this chapter, we will often use the phrase “tree automaton” to mean a “shared 
memory tree automaton” and the phrase “network automaton” ? to mean a “shared 


memory network automaton”. 


In all subsequent chapters, the terms tree and network automaton have different meanings. 
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A closed predicate ? of an automaton A is a predicate L such that for any transition 


(s,7,5) of A, ifs € L then S € L. 


A link subsystem of a tree automaton is an ordered pair (u,v), such that w and 
v are neighbors in the tree. To distinguish states of the entire automaton from the 
states of its subsystems we will sometimes use the word global state to denote a state 
of the entire automaton. For any global state s of a network automaton, we define 
(s|u, s|v) to be the state of the (u,v) link subsystem. Thus the state of the (v,u) link 


subsystem in global state s is (s|v, s|u). 


A local predicate L,,,, of a tree automaton is a subset of the states of a (u,v) link 


subsystem. 


A link predicate set £ for a tree automaton is a set that contains exactly one pred- 
icate for every link subsystem in the tree and which satisfies the following symmetry 
condition: for each pair of neighbors u and v, if (a,b) € Lu», then (b,a) € Ly. (ie., 
while a link predicate set has two link predicates for each pair of neighbors, these two 
predicates are identical except for the order in which the states are written down.) We 
will also assume that every link predicate set is non-trivial in that there is at least one 


global state s such that (s|u,s|v) € Ly» for all link subsystems (u,v) in the tree. 


A tree automaton is locally checkable for predicate L if there is some link predicate 


set £ = {L.,.} such that: 

LD {s:(s|u,s|v) € Ly, for all link subsystems (u,v) in the tree.} 

In other words, the global state of the automaton satisfies L if every link subsystem 
(u,v) satisfies L,,. 


Recall the definition of stabilization in terms of executions and the definition of 
an unrestricted automaton from Chapter 3. Recall that we use A|LZ to denote the 
automaton that is identical to A except that the start states of A|L are the states in 
set L. For any rooted tree T, we let height(T) denote the maximum length of a path 


between the root and a leaf in T. 


We can now state a simple theorem. 


3This is often called a stable predicate. We avoid this phrase because of potential confusion with 


stabilization. 
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Theorem 4.3.1 Tree Correction in Shared Memory Systems: Consider any 
tree automaton T for tree T that is locally checkable for predicate L. Then there exists 
an unrestricted tree automaton Tt for T such that T* stabilizes to the executions of 


T|L in time proportional to height(T). 


Thus after a time proportional to the height of the tree, any execution of the new 
automaton 7+ will “look like” an execution of J that starts with a state in which D 
holds. To prove this theorem we first describe how to construct J* from 7 and then 


show that J* satisfies the requirements of the theorem. 


Assume that 7 is locally checkable for predicate LZ using link predicate set £L = 
{Luv}. We start by defining the set of global states that satisfy all local predicates. 
Let L' = {s : (s|u,s|v) € Ly,» for all link subsystems (u,v) in the tree.}. Clearly 
LDL’. Also because of the non-triviality of the link predicate set, L' is not the empty 
set. To construct T+ from T we do the following: 


e We first normalize all node states in J. Intuitively, we remove all states in the 
state set of a node u that are not part of a global state that satisfies L’. Thus 
Su(T +) := {s|ju:s € L'}. The state set of T+ is just the cross-product of the 
normalized state sets of all nodes. Intuitively, this rules out useless node states 


that never occur in global states that satisfy all local predicates. 


e We retain all the actions of T but we add an extra precondition (i.e., an extra 
guard) to each action a, € A, of T as shown in Figure 4.4. Intuitively, this extra 
guard ensures that a normal action of T is not taken at node wu unless all links 


adjacent to u are in “good states”. All actions of T remain in the same classes 


in T+. 


e We add an extra correction action CORRECT, for every node wu in the tree that 
is not the root. CORRECT, is also described in Figure 4.4. Intuitively, this extra 
action “corrects” the link between node u and its parent if this link is not in 
a “good” state. Each CORRECT, action is put in a separate class with upper 
bound equal to t,. 


We outline a proof of the theorem by a series of lemmas. The first thing a care- 


ful reader needs to be convinced about is that the code in Figure 4.4 is realizable. 
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The state of J* is identical to J except that the state set 


of each node is normalized to {s|u: s € L’} 


MopIFIED ACTION ay, a, € Ay (*modification of action a, in T*) 
Preconditions: 
Exactly as in a, except for the additional condition: 
For all neighbors v of u: (s|u, s|v) € Lu» 
Effects: 


Exactly as in a, 


CorRRECT,, (*extra correction action for all nodes except the root*) 
Preconditions: (parent(u) = v) and ((s|u, s|v) Z Lu») 
Effects: 
Let a be any state in S,,(7*) such that (a, s|v) € Lu,» 
Change the state of node u to a 


Each CorRRECT,, action is in a separate class with upper bound t, 


Figure 4.4: Augmenting T to create T* 
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The careful reader will have noticed that we made two assumptions. First, in the 
CORRECT, action, we assumed that for any link subsystem (u,v) of T* and any state 
b of node v there is some a such that (a,b) € Ly». Second, we assumed that when 
a modified action a, is taken at node wu, the resulting state of node u has not been 


removed as part of the normalization step. 


We will begin with a lemma showing that the first assumption is a safe one. We 


show that the second assumption is safe later. 


Lemma 4.3.2 For any link subsystem (u,v) of T* and for any state a of node u there 
is some b such that (a,b) € Lu». 


Proof: We know that for any state a of v there is some state s € L' such that s|u = a. 
This follows because all node states have been normalized and because L’ is not empty. 


Then we choose b= s|v._ Jf 


The next lemma shows a local extensibility property. It says that if any node u and 
its neighbors have node states such that the links between u and its neighbors are in 


good states, then this set of node states can be extended to form a good global state. 


Lemma 4.3.3 Consider a node u and some global state s of T* such that for all 
neighbors v of u, (s|u,s|v) € Lu». Then there is some global state s' € L' such that 


s'|u = s|u and s'|v = s|v for all neighbors v of u. 


Proof: We create a global state s’ by assigning node states to each node in the tree 
such that for every link subsystem (u,v), the state of the subsystem is in L,,. Start 
by assigning node state s|u to u and s|v to all neighbors v of u. At every stage of the 
iteration we will label a node z that has not been assigned a state and is a neighbor 
of a node y that has been assigned a state. But, by Lemma 4.3.2, we can do this such 
that the state of the subsystem containing z and y is in L,,. Eventually we label all 
nodes in the tree and the resulting global state is in L’. Once again, this is because for 
every link subsystem (u,v), the state of the (u,v) subsystem is in L,,,. The labelling 
procedure depends crucially on the fact that the topology is a tree. J 


To prove the theorem, we will use the Execution Convergence Theorem (3.4.5). 


However, to apply that theorem we have to work with predicates of T+ (i.e., sets of 
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states of J+) and not link predicates of Tt (i.e., sets of states of link subsystems of 
T+). This is just a technicality that we deal with as follows. For each link subsystem 
(u,v), we define the predicate Li,,, = {s: (s|u,s|v) € Luv}. Clearly, L’ =Li,, 


Next consider some u,v, w such that v = parent(u) and w = parent(v). We assume 
that v # nil. (But v may be the root in which case w is nil.) The next lemma states 
an important stability property. It states that if Li, , holds in some global state s of 


T* it will remain true in any successor state of s if either: 


e vis the root OR 


e L’,,, is also true in s. 
Lemma 4.3.4 Consider some u,v,w such that v = parent(u) £ nil and w = parent(v). 
Suppose there is some global state s of T* such thats € Li,,, and(w A nil) +8 € Li. 


Then for any transition (s,7,5), 8 € Li, ,. 


Proof: It suffices to consider all possible actions 7 that can be taken at either u or v 
in state s. It is easy to see that we don’t have to consider correction actions because, 
by assumption, neither the CORRECT, or the CORRECT, action is enabled in state s. 


Consider a modified action a, of T* that is taken at node u. Suppose action a, 
occurs in state s and results in a state §. By the preconditions of action a,, for all 
children z of u, (s|z,s|u) € Ly. But in that case by Lemma 4.3.3 there is some other 
global state s’ € L’ such that: s’|u = s|u, s'|v = s’|v and s'|z = s|x for all children z of 
u. Thus by the third property of a network automaton, the action a, is also enabled in 
s’ and, if taken in s’, will result in some state say 5’. But since L’ is closed, 8’ € L’ and 
hence (8'|w, 8’|v) € Lu. But by the third property of a network automaton, s|u = 8'|u 


and s|v = s'|v. Thus § € Ly. 
The case of a modified action at v is similar. J 


The previous lemma also shows that our second assumption is safe. If a modified 
action is taken at a node w, resulting in state § then 5 € Li, ,, for some v. Thus by 
Lemma 4.3.3 there is some other state 8’ € L’ such that 3’|u = S|u. Thus 8|u cannot 


have been removed as part of the normalization step. 


The next lemma states an obvious liveness property. If Li,,, does not hold in some 


global state of T+, then after at most ¢, time units we will eventually reach some 
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global state § in which L7,,, holds. Clearly this is guaranteed by the correction actions 
(either CORRECT, or CORRECT, depending on whether u is the child of v or vice 


versa) and by the timing guarantees. 


Lemma 4.3.5 For any any (u,v) link subystem and any any execution a of Tt and 
any state s; of a, if s; ¢ Li, then there is some later state s; € Li, that occurs within 


t, time units of s;. 


Proof: Suppose not. Then either CORRECT, (if u is the child of v) or CORRECT, (if 
v is the child of u) is continuously enabled for t, time units after s;. But then by the 
timing guarantees, either CORRECT, or CORRECT, must occur within t,, time after s;, 


resulting in a state in which L’,, (and, of course, Ly,,,) holds. ff 


We now return to the proof of the theorem. First we define a natural partial order 
on the predicates Li,,. For any link subsystem (u,v), define the child node of the 
subsystem to be u if parent(u) = v and v otherwise. Define the ordering < such that 
Di, < Ly, . iff the child node of the (u,v) subsystem is an ancestor (in the tree T) of 
the child node of of the (w,z) subsystem. 


Using this partial order and Lemmas 4.3.4 and 4.3.5, we can apply the Execution 
Convergence Theorem (Theorem 3.4.5), to show that T+ stabilizes to the executions 
of TT|L' in time height(T) -t,. But any execution a of Tt|L’ is also an execution 
of T|L. This follows from three observations. First, since L’ is closed for T, L’ is 
closed for T+. Second, if L’ holds in all states of an execution a of T*|L’, then no 
correction actions can occur in a. Third, any execution of T|L’ is also an execution 
of T|L because L D L’. Thus we conclude that T* stabilizes to the executions of T|L 
in time height(T) - ty. 

This theorem can be used as the basis of a design technique. We start by designing 
a tree automaton T that is locally checkable for some L. Next we use the construction 
in the theorem to convert T into T+. T+ stabilizes to the executions of T|Z even when 


started from an arbitrary state. 


4.3.1 Weakening the Fairness Requirement 


In the previous subsection, we assigned each CORRECT, action to a separate class. Ac- 
tually the theorem only requires a property we call eventual correction: if a CORRECT, 


action is continuously enabled, then a CORRECT, action occurs within bounded time. 
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It is interesting that the protocols in [Dij74, AG90] only require that some enabled 
action in the entire network occur in bounded time. In other words, all the actions in 
the entire automaton can be placed in a single class. How can we ensure the eventual 
correction property in a model in which the only guarantee is that some enabled 
action (in the entire network) will occur in bounded time? To make sure the eventual 
correction property holds in such a model, we need to show that it is impossible to 
remain for an unbounded amount of time in a state in which L,,,, does not hold and 
in which some other action other than CORRECT, is enabled. This property can be 
established* quite easily for the protocols in [Dij74] and [AG90]. 


4.4 Rediscovering Dijkstra’s Protocols 


In this section, we will begin by reconsidering the second example in [Dij74]. This 
protocol is essentially a token passing protocol on a line of nodes with process indices 
ranging from 0 to n — 1. Imagine that the line is drawn vertically so that process 0 is 
at the bottom of the line (and hence is called “bottom”) and Process n — 1 is at the 
top of the line (and called “top”). This is shown in Figure 4.5. The down neighbor 
of Process 1 is Process 1 — 1 and the up neighbor is Process 1 + 1. Process n — 1 and 


Process 0 are not connected. 


Dijkstra observed that it is impossible (without randomization) to solve mutual 
exclusion in a stabilizing fashion if all processes have identical code. To break symme- 
try, he made the code for the “top” and “bottom” processes different from the code 
for the others. 


Dijkstra’s second example is modelled by the automaton D2 shown in Figure 4.6. 
Each process 7 has a boolean variable up;, and a bit z;. Roughly, up; is a pointer 
at node z that points in the direction of the token, and z; is a bit that is used to 
implement token passing. Figure 4.5 shows a state of this protocol when it is working 
correctly. First, there can be at most two consecutive nodes whose up pointers differ 
in value and the token is at one of these two nodes. If the two bits at the two nodes 
are different (as in the figure) then the token is at the upper node; else the token is at 


the lower node. 


4T am grateful to Anish Arora for pointing this out to me. 
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Top (Process n-1) 


0 up = false 

0 up = false 

OF Mp Saalse :- eset seetcot ares Token 
1 up=true 

1 up=true 

1 up = true 


Bottom (Process 0) 


Figure 4.5: Dijktra’s protocol for token passing on a line 


For the present, assume that all processes start with 2; = 0. Also, initially assume 
that up; = false for all processes other than process 0. We will remove the need for 
such initialization below. We start by understanding the correct executions of this 


protocol when it has been corectly initialized. 


A process 1 is said to have the token when any action at Process 7 is enabled. As 
usual the system is correct when there is at most one token in the system. Now, it is 
easy to see that in the initial state only MOVE_UPp is enabled. Once node 0 makes 
a move, then MOVE_UP, is enabled followed by MOVE_UP, and so on as the “token” 
travels up the line. Finally the token reaches node n — 1, and we reach a state s in 
which 2; = x4, fori = 0...n—3 and a,_1 # @pn_2. Also in state 5, up; = true 
for 1 = 0...n—2 and up,_, = false. Thus in state s, MOVE_DOWN,_, is enabled 
and the token begins to move down the line by executing MOVE_DOWN,,_» followed 
by MovE_DowN,_3 and so on until we reach the initial state again. Then the cycle 


continues. Thus in correct executions, the “token” is passed up and down the line. 


In the good states for Dijktra’s second example, the line can be partitioned into two 
bands as shown in Figure 4.7. All bit and pointer values within a band are equal. (Ifa 
node within a band has up; = false we sketch it as a pointer that points downwards). 


All nodes within the upper band point downwards and all nodes within the lower band 
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The state of the system consists of a boolean variable 
up; and a bit z;, one for every process in the line. 


We will assume that up) = true and up,,_, = false by definition 
In the initial state x; = 0 fori =0...n—1 and up; = false fori =1...n—1 


Move_UPp (*action for the bottom process only to move token up*) 
Precondition: #9 = #; and up, = false 


Effect: 29 :=~ 20 


Move_Down,-; (*action for top process only to move token down*) 
Precondition: ¢n-2 # fn-1 
Effects: 


En—1 '= €n-2; 


Move_Up;,1 <i<n-— 2 (*action for other processes to move token up*) 
Precondition: 2; # 2;-1 
Effects: 
@i t= i-1; 


up; := true; (*point upwards in direction token was passed*) 
Move_Downy;, 1 < i < n — 2 (*action for other processes to move token down*) 
Precondition: 2; = 2,41 and up; = true and up;,, = false 


Effect: up; := false; (“point downwards in direction token was passed*) 


All actions are in a single class with upper bound #,,. 


Figure 4.6: Automaton D2: a version of Dijkstra’s second example with initial states. The protocol does 


token passing on a line using nodes with at most 4 states. 


86 


Figure 4.7: In the good states for Dijktra’s second example, the line can be partitioned into 2 bands with 
the token at the boundary. 


point upward. The token is at the boundary between the two bands. If the bit value 
X of the upper band is equal to the bit value Y of the lower band, the token is moving 


downwards; if the two bit values are unequal the token is moving downwards. 


We describe these “good states” of D2 (that occur in correct executions) in terms 
of local predicates. In the shared memory model, a local predicate is any predicate 
that only refers to the state variables of a pair of neighbors. Intuitively, we see that if 
two neighboring nodes have the same pointer value, then their bits are equal; also if a 
node is pointing upwards, then so is its lower neighbor. Thus in a good state of D2 , 


two properties are true for any Process 2 other than 0: 


e If up;_, = up; then z;_1 = aj. 


e If up; = true then up,;_, = true. 


First, we prove that if these two local predicates hold for alli = 1...n—1, then 
there is exactly one action enabled. Intuitively, since up,,_, = false and up) = true, we 
can start with process n—1 and go down the line until we find a pair of nodes 1 andi—1 
such that up; = false and up;_, = true. Consider the first such pair. Then the second 


predicate guarantees us that there is exactly one such pair. The first predicate then 
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guarantees that all nodes 7 <1—1 have x; = 2;_; and all nodes k >1 have z, = 2. 
Thus only one action is enabled. If 2; = x;_; andi—1 + 0 then only MovE_Down;_, 
is enabled. If x; = #;_,; and 1 — 1 = 0 then only MOVE_UPy is enabled. If 2; 4 x;_1 
and i # n—1 then only MOVE_UP; is enabled. If 2; 4 z;_, and i = n — 1 then only 
MovE_DOWN,,_; is enabled. 


A similar argument shows that if there is exactly one action enabled, then both 


local predicates hold for alli =1...n—1. 
Let L be the predicate of D2 which consists of the states of D2 in which exactly 


one action is enabled. It is easy to see that D2 is a tree automaton that is locally 
checkable for ZL. Then we can use Theorem 4.3.1 to convert D2 into a new automaton 


D2* which stabilizes to executions in which there is exactly one token in each state. 


The correction actions we add are once again different from the original actions in 
[Dij74]. However, the corrections actions we add (and consequently the proofs) are 


much more transparent than the original version. 


Dijkstra also described two more stabilizing mutual exclusion protocols, one using 
K-state machines, where K is greater than the number of processes, and a solution 
using 3-state machines. In both solutions the topology is assumed to be a ring (i.e., 
the bottom and top processes are connected). The first protocol is easily seen to be 
locally checkable. Unfortunately it is no longer a tree automaton and hence Theorem 
4.3.1 does not apply directly. However, with some extra work it is possible to derive 
the final protocol as a basic protocol augmented with correction actions that ensure 
that each link predicate becomes true. An even simpler way is to derive the K-state 


protocol using an idea that we call counter flushing (see Chapter 10 and Appendix E). 


Dijkstra’s three state protocol uses a different idea altogether. The protocol is not 
even locally checkable. This protocol seems extremely specific to the ring topology 
used. There are two main ideas. First, tokens are passed up and down a line in 
normal operation just as in the second example. Thus if there is more than one token 
and tokens must keep moving, the tokens must eventually “collide” at some node. 
This node can then destroy one of the tokens. This idea seems limited to a line/ring 
topology. The first idea, however, is not sufficient to detect a situation in which there 
are no tokens. The second idea is to exploit the neighbor relation between the top 
and bottom processes to detect the presence of no tokens. The states of the system 


are such that if there are no tokens, then all processes will have the same state. In 
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normal operation, the states of the top and bottom nodes are always different. Thus 
the absence of tokens can be “suspected” locally if the state of the bottom node is 
not equal to that of the top node. In that case, a token is manufactured. The actual 


protocol can be understood using these ideas. 


4.5 Summary 


Much of the initial work in self-stabilization was done in the context of Dijkstra’s 
shared memory model of networks. Later, the work on local checking and correction 
was introduced [APV91b] in a message passing model. The main contribution of this 
chapter is to show that existing work in the shared memory model can be under- 
stood crisply in terms of local checking and correction. Protocols that appeared to be 


somewhat ad hoc are shown to have a common underlying principle. 


However, as we have argued at the beginning of this chapter, we believe that 
message passing models are more useful and realistic. For the rest of the thesis we will 
concentrate on message passing models. The definitions of network automata, local 
predicates, local checkability, local correctability, and link subsystems that we used in 
this chapter are specific to shared memory systems. In the next chapter (Chapter 5) we 
will introduce definitions of these concepts for networks in which nodes communicate 
by message passing. The definitions in Chapter 5 will be used for the remainder of the 
thesis. 


The main theorem in this chapter states that any locally checkable protocol that 
uses a tree topology can be efficiently stabilized. As the reader might expect, there is a 
corresponding Tree Correction theorem for message passing systems that is described 
at the end of Chapter 6. 
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Chapter 5 


Local Checking and Correction for 


Network Protocols 


In Chapter 4 we introduced a method of local checking and correction using a shared 
memory model taken from [Dij74]. Recall that in such a model, nodes communicate 
with their neighbors by reading the state of all neighbors in one atomic action. Thus 
there is no need to model channels between nodes. The shared memory model allowed 
us to introduce local checking and correction in a fairly simple way. However, it is not 
very realistic because of the high degree of atomicity that it assumes. In this chapter, 
and for the rest of the thesis, we will model communication between nodes by explicit 
message passing through links. However, we will restrict ourselves to a special type of 


link called a Unit Storage Data Link or UDL. 


We begin in Section 5.1 by describing our model of a network protocol. We do so by 
modelling the network topology, the links between nodes by Unit Storage links, and the 
nodes themselves. Our model of a Unit Storage Link is new, and so in Section 5.2 we 
argue that such links can be implemented in real-life networks. Section 5.3 introduces 
the important concept of locality in network protocols: some key concepts such as 
local subsystems, local checkability, and local correctability are defined in this section. 
While many of the ideas are similar to the ideas in Chapter 4 (that were developed 
for shared memory systems), the new definitions are slightly more complex because of 
the presence of channels between nodes. The definitions in this chapter are used for 


the remainder of the thesis. 


In Section 5.4 we state the main result of this chapter, the Local Correction Theo- 
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rem. In essence, this result states that any locally correctable protocol can be stabilized 
using a simple transformation. The transformation involves the addition of extra ac- 
tions to do local checking and correction. Section 5.4 also contains a formal description 
of the transformation. The next two sections contains a proof of the Local Correction 
Theorem. We first provide an intuitive “proof” that presents the main ideas and then 
present a formal proof. The formal proof is important because it shows how we use 
the proof techniques of Chapter 4 to formally prove stabilization results. The proofs 
of the Tree and Global Correction theorems in later chapters are much more intuitive; 
the formal proof in this chapter provides an important example of how such intuitive 


proofs can be formalized. 


Finally, Section 5.7 argues that the method of local checking and correction (that 
is formalized by the Local Correction Theorem) is practical, and can be added to real 


networks without an appreciable loss in efficiency. 


5.1 Modelling Network Protocols 


For the rest of this thesis, we will restrict ourselves to proving stabilization properties 
for network protocols and network automata. To model a network protocol, we need 
to model the network topology, the links between nodes, and the nodes themselves. 
Our model is essentially the standard asynchronous message passing model except for 


three differences: 
e The major difference is that links are restricted to store at most one packet at a 
time. 


e The nodes are restricted to use a certain stylized discipline for sending packets 


on unit storage links. 


e We assume that for every pair of neighbors, there is some a priori way of assigning 


one of the two nodes as the “leader” of the pair. 


We will argue that even with these differences our model can be implemented in 


real networks. 
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5.1.1 Modelling Network Topology 


We use a directed graph G to specify the network topology. For any two neighboring 
nodes u and v, there are two edges (u,v) and (v,u). The network automaton will have 


a undirectional channel corresponding to each directed edge. 


In addition, we require that G satisfies the following property: there is a function 
that for any pair of neighboring nodes u and v in G assigns one of the two nodes as 
the “leader” of the edges (u,v) and (v,w). Thus we are really requiring a way to break 


symmetry between neighboring nodes in G. 


It is possible to remove this assumption (of having a leader function) at the cost of 
some increased complexity in the protocols. However this assumption is not restrictive 
in practice. In a real implementation, if every node has a unique ID then a simple 
stabilizing protocol can elect the minimum ID node as leader. Each node can period- 
ically transmit its ID to its neighbor and both nodes choose the minimum ID. If the 
nodes do not have IDs then an equally simple randomized protocol can elect a leader 
on every link. We prefer to encode the leader function directly in the graph G instead 


of presenting these simple protocols explicitly. 


In summary, the nodes and edges of G correspond to the actual physical topology 
of the network, while the leader function describes a way to break symmetry between 


neighboring nodes. Formally: 


We will call a directed graph (V, FE) symmetric if for every edge (u,v) € EF there is 
an edge (v,u) € E. 


Definition 5.1.1 A topology graph G = (V,E,l) is a symmetric, directed graph 
(V, E) together with a leader function | such that for every edge (u,v) € E, (u,v) = 


I(v,u) and either I(u,v) =u or I(u,v) =v. 


We use E(G) and V(G) to denote the set of edges and nodes in G. If it is clear 
what graph we mean we sometimes simply say EF and V. As usual, if (u,v) € E we 


will call v a neighbor of w. 


The following definition of a leader edge is useful later. Of the two possible edges 


between two neighbors, it produces the edge directed away from the leader. 
Definition 5.1.2 We call (u,v) a leader edge of G if (u,v) € E and I(u,v) =u. 
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5.1.2 Modelling Network Links 


Traditional models of a data link have used what we call Unbounded Storage Data 
Links that can store an unbounded number of packets. Now, real physical links do 
have bounds on the number of stored packets. However, the unbounded storage model 


is a useful abstraction in a non-stabilizing context. 


Unfortunately, this is no longer true in a stabilizing setting. If the link can store 
an unbounded number of packets, it can have an unbounded number of “bad” packets 
in the initial state. It has been shown [DIM91a] that almost any non-trivial task is 
impossible in such a setting. Thus in a stabilizing setting it is necessary to define Data 


Links that have bounded storage. 


A network automaton for topology graph G consists of a node automaton for every 
vertex in G and one channel automaton for every edge in G. We will restrict ourselves 
to a special type of channel automaton, a unit storage data link or UDL for short. 
Intuitively, a UDL can only store at most one packet at any instant. Node automata 
communicate by sending packets to the UDLs that connect them. In the next section, 


we will argue that a UDL can be implemented over real physical channels. 


We fix a packet alphabet P. We assume that P = Pagia U Peontrot consists of two 
disjoint packet alphabets. These correspond to what we call data packets and control 
packets. The specification for a UDL will allow both data and control packets to be 
sent on a UDL. 


Definition 5.1.3 We say that C,, is the UDL corresponding to ordered pair (u,v) 
and with link delay t if Cu 1s the UIOA defined in Figure 5.1. 


By the convention we have established, C,,,, is a UIOA since we have not defined 
any start states for C,,,. The external interface to Cy, includes an action to send a 
packet at node u (SEND,,,(p)), an action to receive a packet at node v (RECEIVE,.»(p)), 
and an action FREE,,, to tell the sender that the link is ready to accept a new packet. 
The state of C,,, is simply a single variable Q,,, that stores a packet or has the default 


value of nil. 


Notice two points about the specification of a UDL. The first is that if the UDL 
has a packet stored, then any new packet sent will be dropped. Second, the FREE 


action is enabled continuously whenever the UDL does not contain a packet. 
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Each p belongs to the packet alphabet P defined above. 


The state of the automaton consists of a single variable Qu, € PU nil. 


SEND,,»(p) (*input action”) 
Effect: 
If Quy = nil then Quy := Dp; 


FREE,» (*output action*) 
Precondition: Qu, = nil 
Effect: None 


RECEIVE,,,(p) (*output action”) 
Precondition: p = Quy 4 nil 
Effect: Quy := nal; 


The FREE and RECEIVE actions are in separate classes with an upper 
bound called the link delay which is equal to ¢ for both classes. 


Figure 5.1: Unit Storage Data Link automaton 
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5.1.3. Modelling Network Nodes 


Next we specify node automata. We do so using a set that contains a node automaton 
for every node in the topology graph. For every edge incident to a node u, a node 
automaton N,, must have interfaces to send and receive packets on the channels cor- 
responding to that edge. However, we will go further and require that nodes obey a 


certain stylized convention in order to receive feedback from and send packets on links. 


In the specification for a UDL if a packet p is sent when the UDL already has a 
packet stored, then the new packet p is dropped. We will prevent packets from being 
dropped by requiring that the sending node keep a corresponding free variable for the 
link that records whether or not the link is free to accept new packets. The sender sets 
the free variable to true whenever it receives a FREE action from the link. We require 
that the sender only send packets on the link when the free variable is true. Finally, 
whenever the sender sends a packet on the link, the sender sets its free variable to 


false. 


We wish the interface to a UDL to stabilize to “good behavior” even when the 
sender and link begin in arbitrary states. Suppose the sender and the link begin in 
arbitrary states. Then we can have two possible problems. First, if free = true but 
the UDL contains a packet, then the first packet sent by the sender can be dropped. 
However, it is easy to see that all subsequent packets will be accepted and delivered 
by the link. This is because after the first packet is sent, the sender will never set free 
to true unless it receives a FREE notification from the link. But a FREE notification 
is delivered to the sender only when the link is empty. The second possible problem 
is deadlock. Suppose that initially free = false but the channel does not contain a 
packet. To avoid deadlock, the UDL specification ensures that the FREE action is 


enabled continuously whenever the link does not contain a packet. 


Thus we will require that each sending node u keep a corresponding free,,|v| variable 
for each neighbor v. By our convention, u can only send packets to v when free, |v] = 
true. Thus we will also require that u enqueue packets that it wants to send to v in an 
outbound queue for the link called queue, |v]. When free, |v] becomes true, the packet 
at the head of the outbound queue is sent on the link. The use of the queuing and 
free disciplines is quite natural for a UDL; more importantly, these conventions make 
it easy to transform a node automaton to do local checking. This will become clearer 


in a few sections. 
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The preceding paragraphs motivate the following formal definition. 


Definition 5.1.4 We say that an automaton N is a node automaton for node u in 


graph G =(V,E,1) and with node delay t if: 


e N has an output action SENDz»(p) and input actions RECEIVE,,,(p) and FREEy,» 
for each p © Pasig and for each neighbor v of u. 


N has a boolean variable free,|v| and a queue variable queue,|v| for every v 


such that (u,v) € E. The queue variable queue,|v] is a queue of bounded size 


consisting of packets drawn from the data packet alphabet Piata-' 


e Actions of N other than SEND,z.»(p) can only change queue,|v| by adding packets 
(drawn from alphabet Piata) to the tail of queue,|v]. Of course, such actions can 


leave the queue unchanged. 


e The code for the output action SENDu,(p) and the input action FREE,, at node 


N is as shown in Figure 5.2. 


e Each SENDu.»(p) action in N is in its own class with upper bound called the node 
delay that is equal to t for all classes. 


In particular, note that every SEND,,,(p) action at a node is a locally controlled 
action. Also, in all the network automata described in this thesis, the transitions 
in automaton N,, will only depend on the state of u, the leader function I, and the 
identities of the neighbors v of u. In other words, N,, will use only local information 
available to wu about the graph G. We prefer not to formalize this requirement as 
part of the definition of a network automaton. Instead, we will use this as an informal 
criterion to rule out trivial solutions (to some problems) in which N,, encodes the entire 


graph. 


1The reader may be puzzled by the fact that the links accept both data and control packets but the 
node automata only send data packets. In a few sections, we will create augmented node automata by 
adding actions to ordinary node automata. These extra actions are used to send and receive control 


packets for the purposes of local checking /correction. 


96 


SENDy,»(p) (*output action to send packet at head of outbound queue™) 


Precondition: free,|v] = true and p is head of queue,,|v] 


Effect: free,|v] := false; Remove p from head of queue,,[v| 
FREEu,» (*input action*) 
Effect: free, |v] := true; (*record that link is free*) 


Figure 5.2: Code at a node automaton to send a data packet and to respond to a free signal from a link 


5.1.4 Network Automata 


Now we are ready to define a network automaton. Naturally, it is the composition of 


a set of node and channel automata. 


Definition 5.1.5 Let G = (V,E,1) be a topology graph. Let N be a set containing 
exactly one node automaton N,,, for every u € V and such that each N,, has node 
delay t. Let Cu, be a UDL for each (u,v) € E such that every UDL has link delay 
t. Then Net(G,N,t,t), the network automaton with node delay t and link delay t, 
is the composition of the automata N,, for allu € V with the automata C,,, for all 
(u,v) EE, 


For the most part we will deal with network automata in which the node and link 
delays are fixed. Thus we let t,, denote the default node delay and t; denote the 
default link delay. If we do not explicitly mention the node and link delays, then 
the node and link delays are assumed to be t, and t; respectively. Thus we will use 
Net(G, N) to denote Net(G, N, tn, t:). We will sometimes say that a time t is a constant 
if ¢ = cyt, + cotj, where c; and cy are some real scalar constants. We use “constant 
time” to emphasize that the time does not depend on the size of the network but only 


on the node and link delays. 
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eee (Pp) Physical Channel RECEIVE (p) 


FREE Data Link Sender Data Link Receiver 


Figure 5.3: Implementing a UDL over a physical channel 


5.2 Implementing Our Model in Real Networks 


In areal network implementation, the physical channel connecting any two neighboring 
nodes would typically not be a UDL. For example, a telephone line connecting two 
nodes can often store more than one packet. The physical channel may also not deliver 
a free signal. Instead, an implementation can construct a Data Link protocol on top 
of the physical channel such that the resulting Data Link protocol stabilizes to the 
behaviors of a UDL (e.g. [AB89], [Spi88a]). 


Figure 5.3 shows the structure of such a Data Link protocol over a physical link. 
The sender end of the Data Link protocol has a queue that can contain a single packet. 
When the queue is empty, the FREE signal is enabled. When a SEND(p) arrives and 
the queue is empty, p is placed on the queue; if the queue is full, p is dropped. If 
there is a packet on the queue, the sender end constantly attempts to send the packet. 
When the receiving end of the Data Link receives a packet, the receiver sends an ack 
to the sender. When the sender receives an ack for the packet currently in the queue, 


the sender removes the packet from the queue. 


If the physical channel is initially empty and the physical channel is FIFO (i.e., 
does not permute the order of packets), then a standard stop and wait or alternating 
bit protocol [BSW69] will implement a UDL. However, if the physical channel can 
initially store packets, then the alternating bit protocol is not stabilizing [Spi88aj. 
There are two approaches to creating a stabilizing stop and wait protocol. Suppose 


the physical channel can store at most X packets in both directions. Then [AB89] 
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suggest numbering packets using a counter that has at least X +1 values. Suppose 
instead that no packet can remain on the physical channel for more than a bounded 
amount of time. [Spi88a] exploits such a time bound to build a stabilizing Data Link 
protocol. The main idea is to use either numbered packets or timers to “flush” the 


physical channel of stale packets. 


A stop and wait protocol is not very efficient over physical channels that have a high 
transmission speed and/or high latency. It is easy to generalize a UDL to a Bounded 
Storage Data Link or BDL that can store more than one packet. For instance, the 
FREE signal for a BDL should be modified to include the number of packets currently 
stored in the BDL. It is also easy to implement a BDL over a physical channel with 
either bounded storage or bounded delay using the techniques described in [AB89] and 
[Spi88a]. We prefer to use a UDL for the rest of this thesis as it provides a simple 
and elegant interface. However, the reader concerned about efficiency should be aware 
that all the protocols in this thesis can be modified (slightly) to work with BDLs. 


Finally, there is one last concern about UDLs. We have seen that real implemen- 
tations will use Data Links that stabilize to a UDL. However, in our model every link 
is assumed to actually be a UDL. Let S be a network in which each link stabilizes to 
the behaviors of a UDL and such that every node automaton is a UIOA. Let S be the 
same network except that every link is replaced by a UDL. Now a UDL is a UIOA and 
hence is suffix-closed. Thus a nice consequence of Theorem 3.5.7 is that if § stabilizes 
to the behaviors in some problem P, then so does S. Thus, to prove a stabilization 
result about S it suffices to prove the stabilization result about S. This is an example 


of why the modularity theorem (Theorem 3.5.7) is important. 


5.3 Locality 


In this section, we reconsider the definitions of local checkability that we introduced in 
Chapter 4. Our new definitions will be slightly more complex because of the presence 


of channels between nodes. These new definitions will be used for the rest of the thesis. 
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5.3.1 Link Subsystems and Local Predicates 


Consider a network automaton with graph G. Roughly speaking, a property is said 
to be local to a subgraph G’ of G if the truth of the property can be ascertained 
by examining only the components specified by G’. For now we will concentrate on 
link subsystems that consist of a pair of neighboring nodes u and v and the channels 
between them. In Chapter 10, we will discuss how our methods can be generalized to 


arbitrary subsystems. 


In the following definitions, we fix a network automaton NV = Net(G,N). 


Definition 5.3.1 We define the (u,v) link subsystem of N as the composition of Nu, 
Cuy, Cou, and N,. 


For any state s of NV: s|u denotes s projected on to node N,, and s|(u,v) denotes 
8 projected onto C,,,. Thus when WN is in state s, the state of the (u,v) subsystem is 
the 4-tuple: (s|u, s|(u,v), s|(v, wv), s|v). 


A predicate L of N is a subset of the states of V. Let (u,v) be some edge in graph 
G of N. A local predicate L,, of N for edge (u,v) is a subset of the states of the (u,v) 
subsystem in VY. We use the word “local” because L,,, is defined in terms of the (u,v) 


subsystem. 


The following definition provides a useful abbreviation. It describes what it means 


for a local property to hold in a state s of the entire automaton. 


Definition 5.3.2 We say that a state s of N satisfies a local predicate Ly, of N iff 


(s|u, s|(u,v), s|(v, x), 8]v) € Lu. 


We will make frequent use of the concept of a closed predicate. Intuitively, a 


property is closed if it remains true once it becomes true. In terms of local predicates: 


Definition 5.3.3 A local predicate Ly, of network automaton N is closed if for all 


transitions (s,7,5) of N, if s satisfies Ly, then so does 5. 


The following definitions provide two more useful abbreviations. The first gives a 


name to a collection of local predicates, one for each edge in the graph. The second, 
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the conjunction of a collection of “local properties”, is the property that is true when 
all local properties hold at the same time. As in Chapter 4, we will require that the 
conjunction of the local properties is non-trivial — i.e., there is some global state that 


satisfies all the local properties. 


Definition 5.3.4 L is a link predicate set for N = Net(G,N) if for each (u,v) € G 


there is some L,,,, such that: 


e If (a,b,c,d) € Luy then (d,c,b,a) € Luu. (t.e., Luy and Ly» are identical except 


for the way the states are written down.) 
e L= {Luy, (u,v) € G} 


e There is at least one state s of N such that s satisfies Ly, for all Lu» € L. 


Definition 5.3.5 The conjunction of a link predicate set £L is the predicate {s : s 
satisfies Ly» for all Lu, € L}. We use Conj(L) to denote the conjunction of L. 


Note that Conj(£) cannot be the null set by the definition of a link predicate set. 


5.3.2 Local Checkability 


Suppose we wish a network automaton N to satisfy some property. An example would 
be the property “all nodes have the same color”. We can often specify a property of 
N formally using a predicate L of NV. Intuitively, V can be locally checked for L if we 
can ascertain whether L holds by checking all link subsystems of NV. The motivation 
for introducing this notion is performance: in a distributed system we can check all 
link subsystems in parallel in constant time. We formalize the intuitive notion of a 


locally checkable property as follows. 


Definition 5.3.6 A network automaton N is locally checkable for predicate L using 
link predicate set L tf: 


e £ is a link predicate set for N and L D Conj(L). 


e Each Ly, € £ is closed. 
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The first item in the definition requires that ZL holds if a collection of local properties 
all hold. The second item is perhaps more surprising. It requires that each local 


property also be closed. 


We add this extra requirement because in an asynchronous distributed system it 
appears to be impossible to check whether an arbitrary local predicate holds all the 
time. What we can do is to “sample” the local subsystem periodically to see whether 
the local property holds. Suppose the network automaton consists of three nodes u, v 
and w and such that v is the neighbor of both u and w. Suppose the property L that 
we wish to check is the conjunction of two local predicates L,, and L,,. Suppose 
further that exactly one of the two predicates is always false, and the predicate that is 
false is constantly changing. Then whenever we “check” the (u,v) subsystem we might 
find L,,, true. Similarly whenever we “check” the (v, w) subsystem we might find Ly 
true. Then we may never detect the fact that DL does not hold in this execution. We 


avoid this problem by requiring that L,,,, and Ly be closed. 


5.3.3 Local Correctability 


The motivation behind local checking was to efficiently ensure that some property L 
holds for network automaton NV. We would also like to efficiently correct MN to make 
the property true. We have already set up some plausible conditions for local checking. 


Can we find some plausible conditions under which N can be locally corrected? 


To this end we define a local reset function f. This is a function with three 
arguments: the first argument is a node say u, the second argument is any state of 
node automaton N,, and the second argument is a neighbor v of u. The function 
produces a state of the node automaton corresponding to the first argument. Let s 
be a state of N; recall that s|u is the state of Ny. Then f(u,s|u,v) is the state of N,, 
obtained by applying the local reset function at u with respect to neighbor v. We will 
abuse notation by omitting the first argument when it is clear what the first argument 


is. Thus we prefer to write f(s|u,v) instead of the more cumbersome f(u, s|u, v). 


We will insist that f meet two requirements so that f can be used for local correc- 


tion (Definition 5.3.7). 


Assume that the property DL holds if a local property L,,, holds for every edge 


(u,v). The first requirement is that if any (u,v) subsystem does not satisfy L,,, then 
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applying f to both u and v should result in making L,,,, hold. More precisely, let us 


assume that by some magic we have the ability to simultaneously: 


e Apply f to N, with respect to v; 
e Apply f to N, with respect to u; 


e Remove any packets stored in channels C,,, and Cy,y. 


Then the resulting state of the (u,v) subsystem should satisfy L,,,. Of course, in a 
real distributed system such simultaneous actions are clearly impossible. However, we 
will achieve essentially the same effect by applying a so-called “reset” protocol to the 
(u,v) subsystem. We will describe a stabilizing local reset protocol for this purpose in 


the next section. 


The first requirement allows nodes wu and v to correct the (u,v) subsystem if Ly, 
does not hold. But other subsystems may be correcting at the same time! Since 
subsystems overlap, correction of one subsystem may invalidate the correctness of an 
overlapping subsystem. For example, the (u,v) and (v,w) subsystems overlap at v. If 
correcting the (u,v) subsystem causes the (v,w) subsystem to be incorrect, then the 
correction process can “thrash”. To prevent thrashing, we add a second requirement. 
In its simplest form, we might require that correction of the (u,v) subsystem leaves 


the (v,w) subsystem correct if the (v,w) subsystem was correct in the first place. 


However, there is a more general definition of a reset function f that turns out to 
be useful. Recall that we wanted to avoid thrashing that could be caused if correcting 
a subsystem causes an adjacent subsystem to be incorrect. Informally, let us say 
that the (u,v) subsystem depends on the (v,w) subsystem if correcting the (v,w) 
subsystem can invalidate the (u,v) subsystem. If this dependency relation is cyclic, 
then thrashing can occur. On the other hand if the dependency relation is acyclic then 
the correction process will eventually stabilize. Such an acyclic dependency relation 
can be formalized using a partial order < on unordered pairs of nodes: informally, the 


(u,v) subsystem depends on the (v, w) subsystem if {v,w} < {u,v}. 


Using this notion of a partial order, we present the formal definition of a local reset 


function: 
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Definition 5.3.7 We say f is a local reset function for network automaton N = 
Net(G,N) with respect to link predicate set L = {Lu} and partial order <, if for any 
state s of N and any edge (u,v) of G: 

e Correction: (f(s|u,v), nil, nil, f(s|v,w)) € Lup. 


e Stability: For any neighbor w of v, 
If (s|u, s|(u,v), s|(v,w), s|v) € Lu, and {v,w} £ {u,v} then 
(slu,s|(u,»),5|(v,u), f(slv,0)) € Law 


Notice that in the special case where all the link subsystems are independent, no 


edge is “less” than any other edge in the partial order. 


Using the definition of a reset function, we can define what it means to be locally 


correctable. 


Definition 5.3.8 A network automaton N is locally correctable to L using link pred- 


icate set L, local reset function f, and partial order < if: 


e N is locally checkable for L using L. 


e f is a local reset function for N with respect to L and <. 


Intuitively, if we have a reset function f with partial order < we can expect the 
local correction to stabilize in time proportional to the maximum chain length in the 
partial order. Recall that a chain is a sequence a, < dy < a3... < a,. Thus the 


following piece of notation is useful. 


Definition 5.3.9 For any partial order <, height(<) is the length of the maximum 
length chain in <. 


5.4 Local Correction Theorem 


5.4.1 Overview 


In the previous section, we set up plausible conditions under which a network automa- 


ton can be locally corrected to achieve a property LD. We claimed that these conditions 
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could be exploited to yield local correction. In this section we make these claims 
precise. We show how to take a network automaton N that can be locally corrected 
to L, and transform it into a new automaton N+. The new automaton N* has the 
property that in all its executions, property LE holds after a bounded amount of time. 
More precisely, V+ stabilizes to the behaviors of M|L in a bounded amount of time. 


The next subsection contains a formal statement of this result. 


To transform N into N+ we will add actions and states to NV. These actions will 
be used to send and receive snapshot packets (that will be used to do local checking 
on each link subsystem) and reset packets (that will be used to do local correction on 
each link subsystem). For every link (u,v), the leader I(u, v) initiates the checking and 


correction.? 


5.4.2 Precise Statement of the Result 


To state the result formally, we need the following definitions. First, when we augment 
a network automaton the resulting automaton should have the same topology and 
also be an unrestricted automaton (UIOA) that can start in any state. The topology 
restriction rules out trivial “centralized” solutions. We also require that the links 


remain UDLs. To formalize these requirements, we define a new type of automaton. 


Definition 5.4.1 Let G = (V,E,1) be a topology graph. An automaton for graph 
G is the composition of an automaton for each u € V, together with Cy, for each 


(u,v) € BE. We assume that all automata being composed are compatible. 


Notice that any network automaton for graph G is also an automaton for graph 
G. However, an automaton for graph G need not be a network automaton because a 
network automaton has additional constraints (such as having outbound queues and 


free variables for each link) on the node automata. 


A UIOA for graph G is an automaton for graph G that is also a UIOA. Recall 
that we used A|L to denote the automaton identical to A except that its start states 
belong to set L. The following piece of shorthand is useful for a concise statement of 


the theorem. 


?Without this assumption, we have to complicate the code and proof to deal with simultaneous 
checking and correction actions by both ends of a link. This can actually be done, thereby getting rid 


of the requirement for a leader on links. But it isn’t worth the increased complexity. 
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Definition 5.4.2 Let N denote a network automaton. We will use N(t) to denote 
the automaton that is identical to N except that the link and node delays in N(t) are 
equal to t. 


Now (finally!) we can state our theorem. Intuitively, it states that if NV is locally 
correctable to LE using local reset function f and partial order <, then we can transform 
N into N* such that N* satisfies the following property: in time proportional to 
height(<), every behavior of V* will “look like” a behavior of NV in which L holds and 


in which the node and link delays are increased by some constant factor. 


Theorem 5.4.3 Local Correction: Consider any network automaton N = Net(G,N) 
that is locally correctable to L using link predicate set £L, local reset function f, and 
partial order <. Then there exists some N+ that is a UIOA for graph G and constants 
c and é such that N+ stabilizes to the behaviors of N(c)|L in time €- height(<). 


5.4.3 Overview of the Transformation Code 


For those familiar with snapshot protocols, the structure of our local snapshot protocol 
is slightly different from the well-known Chandy-Lamport snapshot protocol [CL85]. It 
is easy to show that the Chandy-Lamport scheme cannot be used without modifications 
over unit storage links. Briefly, the reason is as follows. The correctness proof of 
the algorithm in [CL85] is based on reordering executions while preserving causality 
constraints. The only causality constraint for a link in [CL85] is that any action 
that sends a packet p on link EL cannot be reordered to come after an action that 
receives p on link LZ. However, a UDL has an additional causality constraint. A free 
signal delivered by link L after delivering packet p cannot be reordered to come before 
the action that delivers packet p. The Chandy-Lamport scheme was not designed to 
incorporate this extra causality constraint. As a result, if the Chandy-Lamport scheme 
is used unmodified over a network with UDLs, the snapshot may (incorrectly) return 


a state in which there is more than one packet on a UDL! 


Our local snapshot /reset protocol works roughly as follows. Consider a (u,v) sub- 
system. Assume that [(u,v) = u —i.e., u is the leader on link (u,v). A single snapshot 


or reset phase has the structure shown in Fig 5.4. 
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Figure 5.4: The structure of a single phase of the local snapshot/reset protocol 


A single phase of either a snapshot or reset procedure consists of u sending a request 
that is received by v, followed by v sending a response that is received by u. During 
a phase, node u sets a flag (phase,,|v]) to indicate that it is checking/correcting the 
(u,v) subsystem. While this flag is set, no packets other than request packets can be 
sent on link C,,. Since a phase completes in constant time, this does not delay the 


data packets by more than a constant factor. 


In what follows, we will use the basic state at a node u to mean the part of the state 
at u “corresponding” to automaton N,,. To do a snapshot, node wu sends a snapshot 
request to v. A snapshot request is identified by a mode variable in the request packet 
that carries a mode of snapshot. If v receives a request with a mode of snapshot, Node 


v then records its basic state (say s) and sends s in a response to w. 


When u receives the response, it records its basic state (say r). Node u then records 
the state of the (u,v) subsystem as z = (r, nil, nil,s). If e ¢ Lu,» (i.e., local property 


Lu, does not hold) then u initiates a reset. 


To do a reset, node u sends a reset request to v. A reset request is identified by 
a mode variable in the request packet that carries a mode of reset. Recall that f 
denotes the local reset function. After v receives the request, v changes its basic state 
to f(v,s,u), where s is the previous value of v’s basic state. Node v then sends a 
response to wu. When w receives the response, u changes its basic state to f(u,r,v), 


where r is the previous value of u’s basic state. 


Of course, the local snapshot and reset protocol must also be stabilizing. However, 


the protocol we just described informally may fail if requests and responses are not 
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properly matched. This can happen, for instance, if there are spurious packets in the 
initial state of N+. To make the snapshot and reset protocols stabilizing, we number 
all request and response packets. Thus each request and response packet carries a 
number count. Also, the leader u keeps a variable count,|v| that u uses to number all 
requests sent to v within a phase. At the end of the phase, wu increments count,|v]. 
Similarly, the responder v keeps a variable count,|u| in which v stores the number of 
the last request it has received from u. Node v weeds out duplicates by only accepting 


requests whose number is not equal to count, |v]. 


Clearly the count values can be arbitrary in the initial state and the first few phases 
may not work correctly. However, numbering and a few easy checks ensure that in 
constant time a response will be properly matched to the correct request. Because the 
links are unit storage, we will see that a space of 4 numbers is sufficient. Our use of 
numbering is taken from the stabilizing global snapshot protocol of [KP90]. However, 
our protocol is simpler and more efficient because we are restricted to a single link 


subsystem. 


Besides properly matching requests and responses, we must also avoid deadlock 
when the local snapshot/reset protocol begins in an arbitrary state. To do so, when 
phase,,|v| is true (i.e., u is in the middle of a phase), u continuously sends requests. 
Since v weeds out duplicates this does no harm and also prevents deadlock. Similarly, 
v continuously sends responses to the last request v has received. Once the responses 
begin to be properly matched to requests, this does no harm, because u discards such 


duplicate responses. 


An irritating issue that we have to deal with in creating N* is the issue of scheduling 
packets to be sent on links. Notice that the checking and correction protocols are going 
on concurrently with the protocol corresponding to NV. To make Theorem 5.4.3 work, 
we need to ensure that any data packets that are placed on the queue for channel C,,,, 
are sent in constant time. On the other hand, the checking process also needs to send 


request and response packets that are encoded as control packets. 


We build a simple stabilizing scheduler that ensures fair access to the link for each 
of three packet types: requests, responses and data packets. First we notice that at 
the leader end of a link, only requests and data packets need to be sent. At the other 


end, only responses and data packets need to be sent. 


Consider the leader end of a link first. Suppose [(u,v) = u. We know that data 
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packets should never be sent while a snapshot or reset phase is in progress. Now, we 
have a variable phase,,|v| at u which is set to true whenever a phase is in progress. 
The phase ends when a matching response is received and phase,,|v| is set to false. 
At the end of a phase, the oldest data packet is sent, and a new phase is begun after 
the data packet is sent. The scheduler is stabilizing because if there is no data packet 
waiting to be sent, we allow a new phase to begin immediately after the previous phase 
ends. The net effect is that if there are data packets waiting, we send one data packet 
between consecutive checking/correction phases. If a problem is discovered during a 
snapshot phase, we do not do a reset until the next phase, after sending any waiting 


data packet. 


Now consider the end of a link that is not the leader. Suppose I(u,v) = v. To give 
“fair turns” to the response packets we use a variable turn,|v| that has only two values 
data (for data packets), and response (which is the value of turn for response packets). 
No packet can be sent until either its turn arrives or there is no packet of the other 


type. After a packet of a particular type is sent, the turn is “toggled” to the other 
type. 


5.4.4 Constructing Augmented Automata: Formal Descrip- 


tion 


To transform N into N* we will show how to transform each node automaton N,, in 
N into a new, augmented node automaton N,*. Finally, V+ is the composition of the 


new node automata and the (unchanged) channel automata. 


Assume that network automaton NV = Net(G,N) can be locally corrected to L 
using link predicate set L = {L.} and local reset function f. We create Nt = 
Augment(N ,L, f) by adding states and actions to each node automaton N,, as follows: 


e We add the following new variables and their domain specifications, to the state 


of N,,, one for each neighbor v of u as shown below:. 


— count,[v] € {0...3} (used to number request and response packets to ensure 
proper matching. The magic number 3 arises because a link subsystem can 
store at most three distinct counter values on the sending link, receiving 


link, and at the receiver node.) 
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— mode,|v|] € {reset, snapshot} (used to keep track of whether the current 


phase is a snapshot or reset phase.) 


— phase,|v] is a Boolean (set to true during a reset or snapshot phase. It is 
used by the leader node to inhibit data packets from being sent during a 
phase.) 


— freeq,,|v] is a Boolean (set to true after any packet is sent and set to false 
after any packet is removed from the link. The original variable free,,|v| 
will be set to true after a data packet is sent and set to false after a data 
packet is removed from the link. We could have optimized by using just one 
free variable but keeping two variables makes the projection and the proof 


easier.) 


— turn,|v] € {response, data} (used to keep track of whether a data packet or 


a response packet has the next turn to be sent.) 


e We add two new packet types to V*. Recall that there are two basic types of 
packets, data packets and control packets. We will encode request and response 
packets as control packets. We will use the symbols pgata, Preq, ANd Presp to denote 
data, request and response packets respectively. The format of a data packet is 


defined by automaton VV. The encoding of the other two packets is: 


Request : (Control, Request, count, mode ), where count is an integer from 0...3, 


and mode is either reset or snapshot. 


Response : (Control, Response, count, mode, node_state), where count is an in- 
teger from 0...3, mode is either reset or snapshot, and node_state is a state 


of a node automaton N,, in NV. 


As usual, we use record notation to extract fields of a packet. Thus p,.g.count is 


the count field in a request packet. 


e We modify the SEND,,,(p) (for data packets p) and FREE,,, actions of the original 
automaton N,, as shown in Figure 5.5. This code contains modifications for both 
the leader and responder ends in one piece of code; however, some parts of the 


code applies only to leaders and some parts only apply to responders. 


If node u is the leader on link (u,v), then we enable sending data packets only 


when phase,,|v| = false indicates a phase is not in progress. After the data packet 
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SENDy,»(p) (*output action for p € Paata only*) 

Preconditions: 
freeq,,|v] := true and free,,[v] := true 
p is head of queue, |v] 
((U(u, v) = u) and (phase,[v] = false)) OR (([(u, v) = v) and turn,|v] = data)) 

Effect: 
freeq,,|v] := false and free, |v] := false; 
Remove p from head of queue,,|v] 
turn,|v] = response (* give response packets a turn; only affects responders”) 


phase,,[v] = true (* start a new checking/correction phase; only affects leaders*) 


FREE,» (*input action”) 


Effect: freeq,,|v] := true and free,[v] := true; 


Figure 5.5: Code for the modified SEND,,,(p) actions at a modified node N,*. 


is sent, we set phase,|v] = true. If node u is not the leader on link (u,v), then we 
enable sending data packets only when turn,|[v| = data. Also immediately after 
sending a data packet, we set turn,|v] = response, which allows the response 
packets to get a fair turn. We use the freeq, |v] variable to keep track of whether 
there is some packet (either data or control) on C.,,, while free,,[v| keeps track of 
whether there is a data packet on C,,,. The code appears to have some redundant 
checks (for example, the code checks both free variables before sending a data 


packet) but these extra checks do make the proof easier. 


e We add the actions shown in Figures 5.6 and 5.7 to WN, for each neighbor v of u. 
These actions only apply to control packets. We use the following notation. Let 
s denote the current state of V* when an action is performed. Let s|u denote 
the current state of Nt projected onto the original automaton N,,. In order to 
project s to s|w we do the following. All variables of N,, take the same values as 


the corresponding variables in N+. 
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SENDy,v(Preq) (*output action: u repeatedly sends a request till it gets a response™) 


Precondition: 
I(u,v) =u (*u is the leader of link subsystem*) 
(phase,,|v] = true) or (queue, [vu] is empty) (*phase in progress or no data packets waiting 
freeq,,|v] = true (* no packet in transit on link to v *) 
Preq-count = count,|v]; (*count in packet is count of phase*) 
Preq-mode = mode,|v]; (*mode in packet is mode of phase*) 
Effect: 
freeq,,|v] = false (* set to false until link says it is free*) 
phase,,[v] := true; (*remains true until matching response returns*) 
RECEIVEy,u(Preq) (*input action, receive request at u from v*) 
Effect: 
If Preqg. count # count,[v] and I(u,v) = v then (* not a duplicate or invalid packet*) 
count, |v] := Preq. count, (*remember count*) 
mode, |v] := Preq.-mode; (*remember mode*) 


* 


Figure 5.6: Code to send and receive request packets at node w. 


When we apply f to the projected state by setting s|u to f(s|u,v) we affect 
only the projected variables. Thus, for instance, the value of freeq,|v] remains 


unchanged. 


e We add two extra classes for every neighbor v of u to N,. Each output action of 
the form SEND,,,( Control, Request, *) is added to a new partition class. Similarly, 
each output action of the form SEND,,,( Control, Response, *) is added to a new 
partition class. The time associated with all new classes is still the node delay 


tn. 


e We hide all actions of V+ that are not actions of NV. 
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SENDy,v(Presp) (*output action: u repeatedly sends a response to last request*) 
Precondition: 
I(u,v) =v (*u is not the leader of link subsystem*) 
(turn = response) or (queue,,|v] is empty) (*response’s turn or no data packets waiting*) 
freeq,,|v] = true (* no packet in transit on link to v *) 
Presp-count = count,|v]; 
If mode,,[v] = snapshot then presp.node_state = s|u else presp.node_state := f(s|u,v) 
Effect: 


If mode,|v] = reset then s|u := f(s|u, v) (*reset node u’s state locally*) 
mode,,[v] := snapshot (* return to default mode of snapshot*) 
turn,|[v] := data (*give data packets a turn*) 
freeq,,|v] := false (* set to false until link says its free*) 
RECEIVEy,u(Presp) (*input action to receive response at u from v*) 
Effect: 


If (count,[v] = Presp.count) and (phase,,|v] := true) and (I(u, v) = uw) then 
If mode,,|v] = snapshot then 
If (s|u, nil, nil, resp. node_state) ¢ Ly, then mode,|v] := reset 
Else if mode,,[v] = reset then s|u = f(s|u, v) (*reset node u’s state locally*) 
phase,,|v] := false, (*end of phase*) 
count,,|[v] := (count,[v] + 1) mod 4; 


Figure 5.7: Code to send and receive response packets at node uw. 
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5.5 Intuitive Proof of Local Correction Theorem 


In the previous section we described how to transform a locally correctable automaton 
N into an augmented automaton N* We have to prove that in time proportional to 
the height of the partial order every behavior of N+ (described in the last section) is 
a behavior of NV in which all local predicates hold. 


The basic intuition behind the proof is sketched in Figure 5.8 and Figure 5.9. Con- 
sider some (u,v) subsystem in which wu is the leader. We describe the intuition behind 
the use of a counter to ensure proper request-response matching, and the intuition 


behind the local snapshot and reset procedures. 


5.5.1 Intuition Behind Counter Based Matching 


Recall that in the augmented automaton, both local snapshots and local responses are 
implemented using a request-response protocol. As we will see below, both the local 
snapshot and reset procedures will only work correctly if the response from v is sent 
following the receipt of the request at u. The diagram on the left of Figure 5.8 shows 
a scenario in which requests are matched incorrectly to “old” responses that were sent 
in previous phases. Thus we need each phase to eventually follow the structure shown 
in the right of Figure 5.8. 


The code of the augmented automaton ensures that correct matching will occur 
after at most 5 phases by numbering requests and responses. Thus the code uses a 
counter in the range 0..3. The significance of the number 3 will be seen below. The 
sender u keeps a counter count,[v| to number requests and the receiver v keeps a 
counter count,|u| to number responses. Node v accepts a new request numbered c 
only if c # count,|u]. When v accepts this request, v also sets count,[u] to be equal 
to c. Finally, node u accepts a response numbered c only if c = count,|v]. When u 
accepts this response, u also increments count,|v] mod 4. This is implemented in the 


code and illustrated in the diagram on the right of Figure 5.8. 


In the first two phases, packets sent by u and v may be dropped by the links because 
the links may have packets stored in the initial state. However, it is easy to see from 
the properties of a UDL, that after at most two phases, the links in both directions are 


drop-free — i.e., any packets sent from that point on will be delivered. Thus we need 
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Figure 5.8: Using counter flushing to ensure that request-response matching will work correctly within a 


small number of phases. 


to show that within the next three phases, the request-response matching will begin 


to work correctly. This follows from a simple paradigm that we call counter flushing. 


Counter flushing is a general technique that is quite versatile. Chapter 10 describes 
some more applications of counter flushing. In our case, the counter flushing argument 


runs as follows: 


e There can be at most 3 counter values stored in the two links (i.e., the link from 
u to v and the link from v to wu) and the receiver. This is the significance of the 


number 3. 


e The sender retransmits till it gets a response and so the sender counter will keep 


being incremented. 


e Within 3 increments of the sender counter, the sender counter will reach a “fresh” 
counter value that is not present in the links and the receiver. This is because 
the counter space has 4 values, and new counter values can only be created by 


the sender. 


e Suppose the sender sends a request numbered c where c is a fresh value that is not 


present in the receiver or on the two links. Then when a later response numbered 
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c is received, this response is a matching response because no aliasing is possible. 
Also, by the time the sender receives the matching response, the only counter 
value stored in the two links and the receiver is c. In other words, a freshly 
numbered request and its matching response will “flush” the (u,v) subsystem of 


outdated counter values. Hence the term counter flushing. 


e After all old counter values have been flushed, we say that the (u,v) subsystem 
is clean. It is easy to show that all subsequent phases follow the structure shown 


in the diagram on the right of Figure 5.8. 


5.5.2 Intuition Behind Local Snapshots 


The diagram on the left of Figure 5.9 shows why a snapshot works correctly if the 
response from v is sent following the receipt of the request at u. Let a’ and b be the 
state of nodes u and v respectively just before the response is sent. Let a and b’ be 
the state of nodes u and v respectively just after the response is delivered. This is 


sketched in Figure 5.9. 


From the code we know that node u does not send any data packets to v during 
a phase. Also v cannot send another data packet to u from the time the response is 
sent until the response is delivered. This is because the link from v to u is a UDL that 
will not give a free indication until the response is received. Recall that nil denotes 
the absence of any packet on a link. Thus the state of the (u,v) subsystem just before 
the response is sent is (a’, nil, nil, b). Similarly, the state of the (u,v) subsystem just 


after the response is delivered is (a, nil, nil, b’). 


We claim that it is possible to construct some other execution of the (u,v) subsystem 
which starts in state (a’, nil, nil,b), has an intermediate state equal to (a, nil, nil, b) and 
has a final state equal to (a, nil, nil, b'). This is because we could have first applied 
all the actions that changed the state of node u from a’ to a, which would cause the 
(u,v) subsystem to reach the intermediate state. Next, we could apply all the actions 
that changed the state of node v from 6 to b’, which will cause the (u,v) subsystem to 
reach the final state. Note that this construction is only possible because u and v do 
not send data packets to each other between the time the response is sent and until 


the time the response is delivered. 


Thus the state (a, nil, nil,b) recorded by the snapshot is a possible successor of the 
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Figure 5.9: Local Snapshots and Resets work correctly if requests and responses are properly matched. 


state of (u,v) subsystem when the response is sent. The recorded state is also a a 
possible predecessor of the state of (u,v) subsystem when the response is delivered. 
But L,,, is a closed predicate — it remains true once it is true. Thus if L,,, was true 
just before the response was sent, then the state recorded by the snapshot must also 
satisfy D,,. Similarly, if L,, is false just after the response is delivered, then the 
state recorded by the snapshot does not satisfy L,,. Thus the snapshot detection 
mechanism will not produce false alarms if the local predicate holds at the start of the 
phase. Also the snapshot mechanism will detect a violation if the the local predicate 


does not hold at the end of the phase. 


5.5.3 Intuition Behind Local Resets 


The diagram on the right of Figure 5.9 shows why a local reset works correctly if the 
response from v is sent following the receipt of the request at u. Let b be the state 
of node v just before the response is sent. Let a and b' be the state of nodes u and v 


respectively just before the response is delivered. This is sketched in Figure 5.9. 


The code for an augmented automaton will ensure that just after the response 
is sent, node v will locally reset its state to f(b,w). Similarly, immediately after 


it receives the response, node u will locally reset its state to f(a,v). Using similar 
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arguments to the ones used for a snapshot, we can show that there is some execution 
of the (u,v) subsystem which begins in the state (f(a, v), nil, nil, f(b,w)) and ends in 
the state (f(a, v), nil, nil, b’). But the latter state is the state of the (u,v) subsystem 
immediately after the response is delivered. But we know, from the correction property 
of a local reset function, that (f(a, v), nil, nil, f(b, u)) satisfies L,,. Since Ly, is a closed 


predicate, we conclude that L,,, holds at the end of the reset phase. 


5.5.4 Intuition Behind Local Correction Theorem 


We can now see intuitively why the augmented automaton will ensure that all local 
predicates hold in time proportional to the height of the partial order. Consider a 
(u,v) subsystem where {u,v} ¢ {w,x} for any pair of neighbors w,z — i.e., {u,v} is 
a minimal element in the partial order. Then, within 5 phases of the (u,v) subsystem 
the request-response matching will begin to work correctly. If the sixth phase of the 
(u,v) subsystem is a snapshot phase, then either L,,, will hold at the end of the phase 
or the snapshot will detect a violation. But in the latter case, the seventh phase will 


be a reset phase which will cause L,,,, to hold at the end of the seventh phase. 


But once L,,, remains true, it remains true. This is because L,,, is a closed 
predicate of the original automaton N and the only extra actions we have added to Nt 
that can affect L,,, are actions to locally reset a node using the reset function f. But by 
the stability property of a local reset function, any applications of f at u with respect 
to some neighbor other than v cannot affect L,,. Similarly, any applications of f at v 
with respect to some neighbor other than u cannot affect L,,. Thus in constant time, 
the local predicates — corresponding to link subsystems that are minimal elements in 


the partial order — will become and remain true. 


Now suppose that the local predicates for all subsystems with height < 7 hold from 
some state s; onward. By similar arguments, we can show that in constant time after 
8;, the local predicates for all subsystems with height 1 + 1 become and remain true. 
Once again, the argument depends crucially on the stability property of a local reset 
function. The intuition is that applications of the local reset function to subsystems 
with height < z do not occur after state s;. But these are the only actions that can 
falsify the local predicates for subsystems with height 1+ 1. The net result is that all 
local predicates become and remain true within time proportional to the height of the 


partial order <. 
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5.6 Formal Proof of Local Correction Theorem 


The reader who is satisfied with the “intuitive proof” given above should skip this 
section. However, there are a number of details glossed over in the intuitive proof that 
are spelled out in the formal proof. The formal proof also provides a good example 
of the proof techniques of Chapter 3. Formal proofs for other major theorems in 
this thesis like the Tree Correction theorem (Chapter 6) and the Global Correction 


Theorem (Chapter 8) can be constructed along similar lines. 


5.6.1 Overview of Formal Proof 


We wish to prove the local correction theorem, Theorem 5.4.3. Thus we have to prove 
that in time proportional to height(<), every behavior of N* will “look like” a behavior 
of N in which L holds and in which the node and link delays are increased by some 


constant factor. 


The formal proof is based on the proof technique described in Chapter 3 in Lemma 3.4.2. 


Thus the proof consists of two major parts: 


e We first define a predicate @ (more details below) and show that any execution 
of N* stabilizes to the executions of V*|Q. We prove this using the Execution 


Convergence theorem, Theorem 3.4.5. 


e We show that any behavior of N’*|Q is also a behavior of M(c)|L, for some 
constant c. We prove this using the Refinement Mapping theorem, Theorem 


3.4.3 


We now present a more detailed roadmap of each part of the formal proof. 


The first part of the proof is described in Sections 5.6.2 to 5.6.4. To show that N* 


stabilizes to the executions of Nt|Q we use another two step process: 


e In Section 5.6.3 we formally define the concept of a clean link alluded to earlier. 
Intuitively, a link is clean if the snapshot numbering scheme is working correctly, 
meaning that requests and responses will properly be matched. We use the 
predicate C to denote the fact that all links are clean. At the end of Section 
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5.6.3, in Theorem 5.6.14 we prove that Nt stabilizes to the executions of N*|C 
in constant time. The proof is based on the counter flushing intuition we have 


described earlier. 


e In Section 5.6.4 we define another important concept called a quiet link. Recall 
that our intent is to make the local predicate L,,,, hold for every link. Intuitively, 
a link (u,v) is quiet if two properties hold. First, L,, holds. Second, if all links 
less than (u,v) in the partial order are also quiet, then there will be no more 
reset actions on that link. We use the predicate @ to denote the fact that all 
links are quiet. At the end of Section 5.6.4, in Theorem 5.6.22 we prove that 
N*|C stabilizes to the executions of N*|Q in time proportional to the height of 
the partial order. 


The second part of proof (Section 5.6.5, (Lemma 5.6.23)) shows that any behavior 
of N*|Q is also a behavior of N(c)|L, for some constant c. This is done using the 
Refinement Mapping theorem, Theorem 3.4.3. In order to use this theorem we must 
derive a projected state of V from a state of N+. We have already seen how to derive 
a projected state s|u of a node N,, from a state s of N;*. To complete our job, we need 
to state how to project the state of the channels in Nt to N. 


Consider the problem of projecting the channel state. Intuitively, in M*, the 
original automaton N is “sharing” each channel with request and response packets. 
Let’s look at the behavior of V*. When a control packet is on the channel C.,,,, we can 
pretend that, as far NV goes, the channel is really empty. There are two consequences 
of this. First, if the projected N,, has a data packet to send, it can take longer before 
the data packet is sent. Second, it can take longer before a free signal is delivered 
by the link. This can happen if there is a control packet on the link. We model this 
by saying that in the projected behavior the node and link delays are increased by a 
constant factor. This is the source of the increased link and node delays in Theorem 


5.4.3. 


Definition 5.6.1 For any state s of N+, we define Proj(s), the state of N* projected 
onto N, as follows: 


e For any two neighboring nodes u and v: if s.Quy © Paata then Proj(s).Quy» = 
8.Quy; else Proj(s).Quy, = nil. (i.e., if there is a data packet p in channel Cy,» 


120 


in the original state, then p is also present in the projected state; if not, Cy, 1s 


considered empty in the projected state. ) 


e All other variables of N have the same values in state Proj(s) as in state s. 


We complete the second part of the proof (Lemma 5.6.23) using the Refinement 
Mapping theorem and using the mapping function Proj. Notice that is not as simple 
as it might appear. If all actions of N* that change the projected state are actions 
of N, then it would be quite easy. Unfortunately we have a complication. There are 
actions of ’* that can cause a local reset. An example is the receipt of a Dresp packet 
with Presp.-mode = reset. Such actions change the projected state but are not actions 
of VN’. However, we will show that such actions cannot appear in executions of VT|Q. 
This is because @ is a strong enough predicate to ensure not only that property L 


holds in the projected state, but also that no local reset actions are enabled. 


We continue to let s|u denote the current state of NV’ projected onto the original 
automaton N,. We also use s|(u,v) to denote the state of Cy, projected onto the 


original automaton N. In other words s|(u,v) = Proj(s).Quy- 


5.6.2 Phases 


We formally define a phase (see Figure 5.4) on a link, as an interval during which 
the checking/correction procedure works on that link. As we have seen, the check- 
ing/correction procedure works by setting phase, |v| = true, and attempting to send 
out a numbered request packet. If a response is received with a matching number then 


phase,,|v] is set to false. We make this more precise below. 


Definition 5.6.2 A (u,v) phase for execution a is any interval y of a such that 


eu=I(u,v) 


e ¥ begins with a state s; of a such that s;.phase,|v| = false and s;,;.phase,|v] = 


true. 


e y ends with the first state s;,7 >% of a in which s;.phase,,|v| = false. 


121 


Notice that a (u,v) phase is only defined for a leader edge (u,v). 
The definition of a phase allows the last state of a phase to overlap the first state 


of the next phase. Clearly we can divide an execution a into consecutive (u,v) phases 
as well as intervals that lie between (u,v) phases. Thus we can speak of the i-th (u,v) 


phase in a@ in this division. 


Our first lemma states that in between phases, at most one data packet is sent on 
Cu. Thus we can think of an execution (from the point of view of any link C,,,,) as 
alternating between a phase during which requests are sent and responses are received, 


followed by a period where at most one data packet is sent on C,,,. 


Lemma 5.6.3 Between two consecutive (u,v) phases on link (u,v) at most one 


SENDu»(Pdata) event can occur. 


Proof: The first SEND.» (Pdata) event after the i-th (uw, v) phase will set phase,,|v| = true 
which begins the 1+ 1-st phase. J 


The proofs of the next two lemmas are quite tedious and are relegated to the 
appendix. The first lemma states that a single phase completes in constant time. 
Intuitively, the time taken to complete a phase consists of the time to start a phase 
(which may involve the sending of a data packet) followed by the time to send a 
request and receive a matching response. The second lemma shows that data packets 
are sent out on a link in a bounded time after they are placed on the queue for the 
link. Intuitively, this is because a phase takes constant time to complete and if the 
data queue is non-empty, the code sends at least one data packet between consecutive 


phases. 


Before we state these lemmas, we define a quantity ¢,. Intuitively, t, is the the 


time it takes to complete a phase. 
Definition 5.6.4 We let t, = 6t, + 12t;. 


The major components of the time to complete a phase are sketched in Figure 5.10. 
Since the free variables may be incorrect in the initial state, the first packet sent on 
a link can be dropped. However, in constant time, the links stop dropping packets. 
Then, after the possible sending of data packet, a request is sent out by u in constant 
time. Next, after the possible sending of a data packet, a response is sent by v in 


constant time. The appendix contains more details. 
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Figure 5.10: The major components of the time required to complete a phase. 


Lemma 5.6.5 Phase Rate: For all a and any leader edge (u,v) and any integer z, 


at least x (u,v) phases will have completed within (cx +1)-t, time units after the start 


of a. 


Proof: From Lemma B.3.4 and Lemma B.3.5 in Section B.3 of the appendix. The 
reason why we need (z + 1)-¢, time (instead of x -t, time) to complete z phases is as 
follows. Suppose in the initial state of a, phase, |v] = true; it may take ¢, time before 
phase,,|v| becomes false. Thus, the first (u,v) phase in a may be a “partial phase”. 
However, the definition of a phase does not allow us to consider a “partial phase” as 


aphase. [ff 


Lemma 5.6.6 Data Packet Rate: For any a = s9,a1,... and any (u,v), if 


S9.|queue,|v]| > 0 then a SENDy»(Pdata) occurs within t, time units after so. 


Proof: At the end of Section B.3 in appendix. J 


5.6.3 Clean Phases 


We know that a link can drop packets if a packet is sent when the link already has a 
packet. To be sure that packets will not be dropped, the sender must never think that 
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count [v] is the count at sender count | [u] is the count at receiver 
u 


reqcount(u,v) is the count carried in a request 


respcount(u,v) is the count carried in a response 


phase aid is set to true 
during a phase 


Figure 5.11: The key variables used in the definition of a clean link: countset(u, v) is the union of the 
count at the receiver and any count values in request and response packets. A clean link ensures that 


between phases, the sender count value is not in countset. 
a link is free when it isn’t. In this case, we say the link is “drop-free”. 


Definition 5.6.7 Let F,,, denote the predicate of N* defined by: (freeq,,[v] = true) > 
(Qu = nil). We also say that (u,v) is drop-free in state s of Nt ifs © Fuy. 


In the appendix, we show that once a link is drop-free, it remains drop-free. In- 
tuitively, this is because the sender will not record the link as free unless it receives 
a free signal from the link, which means that there is no packet on the link. In the 
appendix, we also show a link (u,v) becomes drop-free in constant time — in fact, after 
the first packet sent on the link. This follows because after the first packet is sent 
on the link, the sender records the link as being busy, and this trivially satisfies the 


drop-free predicate. 


Before studying the structure of phases, we introduce some notation to denote the 
set of counter values in a (u,v) subsystem. These include counters in any request or 
response packets, and the counter stored at node v. These variables are sketched in 


Figure 5.11. 


Definition 5.6.8 For any state s we define the derived variables reqcount(u,v), 


respcount(u,v) and countset(u,v) as follows: 
1. If Qu = Preq then reqcount(u,v) = Preq-count; otherwise reqcount(u,v) = undefined 
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2. If Quu = Presp then respcount(u,v) = Presp.count; otherwise respcount(u,v) = 


undefined. 


3. countset(u,v) is the set formed by the union of the values in respcount(u, v), 


regcount(u,v) and count, |u| 


In order for a (u,v) phase to work correctly, we need the phase to follow the 
structure describe in Figure 5.4. For this to happen, when a numbered request is 
first sent during a phase, the number of the request should not already present at 
the receiver or in the channels. If this is not the case, it is possible for an incorrect 
response to be accepted in the phase. We formalize this notion of a link and a phase 
being “clean” using five conditions. The second condition is the crucial condition; 
the other four are supporting conditions required to ensure that the clean predicate is 


closed. 


Definition 5.6.9 We say that a leader edge (u,v) is clean in state s of N* iff all the 


following predicates are true in s: 
1. (v,u) and (u,v) are drop-free. 
2. If phase, |v] = false then count,|v| ¢ countset(u,v) 
3. If phase, |v] = true then reqcount(u,v) = count,|v] or reqcount(u,v) = undefined 
4. If phase,|v] = true and respcount(u,v) = count,|v| then respcount(u,v) = count, |u]. 
5. If count,|u] = count,|v] then Quy ¢ Paata- 
The first condition ensures that the links in both directions are drop-free. The 
second condition ensures that when a numbered request is first sent during a phase, 


the number of the request is not already present at the receiver or in the channels. 


This is the important property of a clean link. 


The third condition states that during a phase, any requests must carry the sender’s 
number. The fourth condition states that during a phase if there is a matching response 
on the channel from the receiver to the sender, then the receiver has the same number 


as the sender. The fifth condition says that if during a phase the sender and receiver 
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numbers are the same, then there can be no data packet in transit from the sender to 
the receiver. The fifth condition (as we will see below) follows from the fact that the 


sender will not send a data packet during a phase. 
Definition 5.6.10 A (u,v) phase p is clean if (u,v) is clean in the first state of p. 


Intuitively, a clean phase will contain an action to send a request packet at, u 
followed by the receipt of this packet at v followed by the sending of a response by 
u followed by the receipt of the response at u. The receipt of the matching response 
ends the phase. This is shown in Figure 5.4. Thus a clean phase will ensure that the 


response received at node u will correspond to the requesting information at node wu. 


A nice property is that once an edge becomes clean, it remains clean. 


Lemma 5.6.11 For any transition (s,7,8) and any leader edge (u,v), if (u,v) is clean 


in s then (u,v) is clean in 8. 


Proof: See Section B.4 in the appendix. However, it is not hard to see informally see 
why this is true. First, as we have seen before, once a link is drop-free, it remains 
drop-free. Next, we know from the fourth condition that a matching response can 
only be received if the receiver has the same number as the sender. Also from the 
second condition, any requests present during a phase must have the same number as 
the sender. Now the sender always increments its number after receiving a matching 
response. Thus after a phase is over, the number of the sender must be different from 
any of the numbers present in the receiver or channels: this is the second condition. 
The third condition follows from the fact that a phase begins by sending a request 
with the sender’s number, and subsequent retransmissions of requests carry the same 
number. Next, if a matching response is present, it must have been sent by the receiver 
during the phase. But, by the third condition, the receiver cannot change its number 
after it sent the response. Thus if there is a matching response, the number of the 


receiver must be the same as the sender, which is the fourth condition. 


The fifth condition follows because the sender never sends a data packet during 
a phase. By the second condition, at the start of the phase the receiver number is 
not equal to the sender number; thus the two numbers become the same only after 


the receiver has received a request. But the receipt of this request “flushes” out any 
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data packets that were in transit from sender to receiver at the start of the phase. 
This (taken together with the fact that the sender never sends a data packet during a 
phase) is what makes the fifth condition hold. J 


The following lemma explains why the request-response matching procedure is 


stabilizing. 


Lemma 5.6.12 A leader edge (u,v) is clean in all states after the fifth (u,v) phase 


in any execution a. 


Proof: (Idea) We first show that after at most two phases (v,u) and (u,v) are both 
drop-free. This follows (see Claim B.2.2 in the appendix) since some packet (i.e., at 
least one request and at least one response) must have been sent on either link by 
this time. Next consider the end of the second phase. In this state, there must be 
some c € {0,...,3} such that c ¢ countset(u,v). This follows because, by definition, 
countset(u,v) has a maximum of three elements. Now consider the first time after the 
end of the second phase that count,|v] = c. This must occur at the end of a phase 
because count,|v| is only incremented at the end of a phase. Also this must occur at 
or before the end of the fifth phase because count,|v| increases by 1 mod 4 at the end 
of each phase. 


It is easy to see using Claim B.4.1 that when count,|v] first becomes equal to c, 
c ¢ countset(u,v). Thus at or before the end of the fifth phase, phase, |v] = false and 
count,|v| ¢ countset(u,v). Thus at or before the end of the fifth phase, edge (u,v) is 


clean. Finally, by Claim 5.6.11, (u,v) is clean in all subsequent states ofa. I 


Now we have reached the goal of this subsection. First we define: 


Definition 5.6.13 Let C be the predicate of N* such that for all leader edges (u,v), 


(u,v) is clean in all states in C. 


Recall that Nt|C is the automaton that is identical to V+ except that all leader 


edges are clean in any initial state. 


Theorem 5.6.14 Nt stabilizes to the executions of Nt|C in time 6ty. 
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Proof: Follows directly from Lemma 5.6.11, Lemma 5.6.12 and Lemma 5.6.5 and the 
Execution Convergence theorem, Theorem 3.4.5. Lemma 5.6.11 shows that each leader 
edge becomes clean after at most 5 phases which by Lemma 5.6.5 takes at most 6¢, 
time. Lemma 5.6.11 shows that once a leader edge is clean it remains clean. Then the 
Execution Convergence theorem (Theorem 3.4.5) shows that all leader edges become 


clean in at most 6t, time. J 


5.6.4 Quiet Links: Establishing Link Predicates 


Clean phases are only useful because they allow local predicates to be established as 
we show in this section. We know from the last section that every execution has a 
suffix that is clean. When we say that L,, holds in a state s of N+, we mean that 
(s|u, s|(w,v), s|(v, uw), s|v) © Lu». 

Recall that < is the partial order on edges associated with reset function f. Con- 
sider a link (u,v) such that there is no other link {w,xz} < {u,v}. We will show that 
if in the first state of a clean (u,v) phase mode,|v|] = snapshot then at the end of the 
phase, a true snapshot of the (u,v) subsystem is obtained. In particular, it is possible 
at the end of such a phase to determine whether L,,,, holds at the end of the phase. If 
it does not hold, mode,,|v] is changed to reset. In a similar fashion, we can show that 
any clean phase whose initial state has mode,|v] = reset will guarantee that L,,, holds 
at the end of the phase. The net effect is that L,,, holds by end of the second (u,v) 


phase of a “clean” execution. 


However, we want to show not only that L,, holds by end of the second (u,v) 
phase but also that no more “reset” actions can occur after this point so that L,,, will 
remain true. This motivates the following definitions of a quiet link. Please refer to 


Figure 5.12 for an intuitive explanation. 


Definition 5.6.15 We say that a leader edge (u,v) is quiet in state s of Nt iff the 
following predicates hold in s: 


1. (u,v) ts clean. 
2. (slus|(u,~),5|(v,), s/o) € Law 
3. (mode,|v|] # reset) and (mode,|u] # reset). 
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Leader mode should Responder mode should 


not be reset not be reset 


Any Request that is not outdated must not be a reset request 


ae 
Any Response will not cause leader to change mode to reset 


LEADER 


Figure 5.12: A leader edge is quiet if it clean, its local predicate holds, and it satisfies the conditions 
sketched in the figure. 


4. If Quw = Preq aNd Preq.mode = reset then preq.count = count, |u]. 


5. If Quu = Presp GN Presp.count = count,|v] then (s|u, nil, nil, Dresp.node_state) € 


Laks 


Notice that the fourth condition above ensures that there is no reset request on the 
link that could be accepted by the receiver at a later state. The fifth condition ensures 
that any snapshot information in a matching response will not cause the sender to 


change its mode to reset. 


Our goal is to show that eventually all links are quiet. 


Definition 5.6.16 Let Q be the predicate set consisting of the following predicate for 
each leader edge in G. The predicate for each leader edge (u,v) is that (u,v) is quiet. 
Let Q be the predicate that is the intersection of all predicates in Q. 


We will extend the partial order < (which was defined on undirected pairs of 
nodes) to leader edges by assuming that that leader edge (u,v) < leader edge (w,z) 
iff {u,v} < {w,az}. Next, we will use the Execution Convergence theorem (Theorem 
3.4.5) to show that M* stabilizes to the executions of N*+|Q. We know that Nt 
stabilizes to the executions of Nt|C. Thus it is sufficient to show that N*|C is 
stabilized to @ using predicate set Q and partial order <. To show this, we will first 


show two lemmas: 
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First we state the required stability condition: that a leader edge (u,v) will remain 


quiet as long as all leader edges less than or equal to (u,v) are quiet. 


Lemma 5.6.17 Consider a leader edge (u,v). If every leader edge (w,z) < (u,v) is 
quiet in some state s of N*|C, then for any transition (s,7,8), (u,v) is quiet in state 


s. 


Proof: In Section B.5 in the appendix. However, the basic idea is simple. We have 
already seen that the clean predicate is stable. We also know from the definition of 
local checkability that L,,., is closed for all the actions of the original automaton WV. 
However, in Nt we added additional actions to send requests and responses. It is 
easy to verify that the sending of snapshot requests and responses do not affect Ly». 
However, L,,, is affected by actions that send reset requests and responses on edges 
to neighbors that are less than (u,v) in the partial order. (If such “reset” actions 
occur on edges to neighbors that are not less than (u,v) in the partial order, then the 
definition of the local correction function ensures that L,, remains true.) But such 
“reset actions” cannot occur on edges less than (u,v) in the partial order, because 


such edges are quiet by hypothesis. 


Next, if LZ, holds, and the snapshot information in matching responses is always 
correct, the sender will never change its mode to reset. But if the sender never changes 
its mode to reset, the sender will never send a reset request. And if the receiver never 


receives a reset request, the receiver will never change its mode to reset. Jf 


Next we state the required liveness condition: that a leader edge (u,v) will become 


quiet in bounded time if all leader edges less than (u,v) are already quiet. 


Lemma 5.6.18 Consider a leader edge (u,v) and some execution a of N*|C. If every 
leader edge (w,z) < (u,v) is quiet in some state s; of a, then (u,v) is quiet in some 


state that occurs within 3-t, of 5; in a. 


Proof: In In Section B.5 in the appendix. However, the basic idea is simple. Consider 
the first complete (u,v) phase after s;. If it is a snapshot phase, and L,,, does not hold 
at the end of the phase, this will be detected and the next phase will become a reset 
phase. Thus either the first or second complete phase after s; will be a reset phase. 


At the end of the reset phase, (u,v) will become quiet because all the links that (u,v) 
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depends on are already quiet. The upshot is that within two complete phases, (u,v) 
becomes quiet. But this can take up to 3t, time (where ¢, is the time to complete a 


single phase) because the first phase after s; can be an incomplete phase. J 


The last two lemmas immediately give us: 
Lemma 5.6.19 N*|C is stabilized to Q using predicate set Q and time constant 3tpy. 


Thus applying the Execution Convergence Theorem (3.4.5) and using the last 


lemma, we get: 
Lemma 5.6.20 N*|C' stabilizes to the executions of N*|Q in time height(f) - 3tp. 


For convenience we define a quantity ¢, which intuitively can be thought of as the 


time after which any execution of N+ becomes quiet. 
Definition 5.6.21 We let tz = 6t, + height(f) - 3tp; 
Thus we have the major result of this section: 
Lemma 5.6.22 N7* stabilizes to the executions of Nt|Q in time tg. 


Proof: Follows directly from Lemmas 5.6.14 and 5.6.19 and transitivity (Lemma 3.1.6). 
| 


5.6.5 Projecting Behaviors of N|Q 


In this subsection, we prove that if all leader edges are quiet in the initial state of 
some execution a of Nt, then the external behavior corresponding to a is a behavior 
of N|L. Intuitively, this is not very hard to believe. This is because for every leader 
edge (u,v), Lu» holds in every projected state of a. We also know that there will be 


no reset transitions in a. Formally: 


Lemma 5.6.23 Every behavior of N*|Q is a behavior of N(ty)|L 
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Proof: Follows by using a refinement mapping (Theorem 3.4.3) using the mapping 
function Proj (Definition 5.6.1). 


First, for any state s of Nt|Q, and any leader edge (u,v), we know (from the 
definition of Q) that (s|u,s|(u,v),s|(v,u),s|v) € Lu». Thus Proj(s) € L. Thus for 
any state s of N*, Proj(s) is a start state of N(t,)|L 


Next consider any transition, (s,7,5) of Nt|Q. We know from the definition 
of Q that there are no reset transitions in N*+|Q. Suppose 7 is a SENDu»o(Preq) ; 
RECEIVEn,»(Preq) » SENDv,u(Presp) » Of RECEIVEy,u(Presp) for any (u,v). Then it is easy 


to see that Proj(s) = Proj(s). But such actions are not actions of V. 


If 7 is a SEND,» (Pdata) action for some edge (u,v), then it must be that paata is at 
the the head of s.queue,,[v] and s.free,|v] = true. Since Proj(s).queue,|v] = s.queue,,|v| 
and Proj(s).free,|v] = s.free,|v|, the SENDu»(Pdata) event is also enabled in Proj(s). 
Also, §.Quyv = Proj(s).Qu» = Pdata, since (u,v) is drop-free in s. Thus (Proj(s), 
SENDu,»(Pdata) , Proj(S)) is a transition of V(t,)|LZ. Similarly if 7 is a RECEIVE,» (Pdata) 
action for some edge (u,v), then it must be that s.Qu. = Paata and §.Qu,y = nil. 
Thus Proj(s).Quv = Pdata and Proj(s).Quy = nil. and (Proj(s), RECEIVE:,»(Pdata) ; 
Proj(8)) is a transition of V(t,)|L. Next if 7 is a FREE,,. action for some edge (u,v), 
then it must be that s.Qu, = nil and 5.Qu, = nil. Thus Proj(s).Qu, = nil and 
Proj (8).Qu, = nil and (Proj(s), FREE,,,, Proj(%)) is a transition of N(t,)|L. 


Finally suppose that a is any other action of N(t,)|L. (For example, this category 
would include internal actions of N(¢,)|LZ.) Then such actions remain unmodified and 
the values of all node variables are identical in s and Proj(s). Thus (Proj(s), 7, Proj (8)) 
is a transition of V|L(t,). 


Finally consider the timing properties. Suppose a SEND,z,»(Pdata) event is enabled 
in Proj(s) for any edge (u,v). Then s.|queue,|v|]| > 0. Thus by Lemma 5.6.6, a 
SENDu»(Pdata) event occurs within t, time after s. Suppose a RECEIVE,» (Pdata) event 
is enabled in Proj(s) for any edge (u,v). Then s.Qu. = Pdata # nil. From the timing 
properties of N*|Q we know that a RECEIVE,,»(Pdata) will occur in ¢; time, and t; < ty. 


Consider a FREE,,, action for any edge (u,v). Within ¢; time units after s, either 
a FREE,, action will occur or there is a state s’ such that s’.Q., 4 nil. Consider 
the second case. From Claim B.3.1, within t; time units after s’, there is a state s” 
such that s”.Qu, = nil. Since (u,v) is drop-free in all states of a, s” freeq,, |v] = false. 


Thus Qu,» = nil will remain true until a FREE,,,, action occurs within time ¢; after s”. 
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Thus a FREE, action will occur within 3t; time units after state s. In particular, this 
means that a FREE, action will occur within ¢, time units of any state s such that 
FREE, is enabled in Proj(s). 


Finally suppose that 7 is any other locally controlled action of N(t,)|L. Thus 7 
is an internal action of N(t,)|L. Notice that a is in the same class say c (with upper 
bound equal to ¢,) in both N*|Q and N(t,)|L. Suppose that 7 is enabled in Proj(s). 
Then z is enabled in s. Then in ¢, time of s in a either some action in class c occurs or 
there is some state § such that no action in class c is enabled in Proj(). This follows 
because if no action in class c is enabled in §, then no action in class c is enabled in 


Proj(&), and vice versa. [J 


5.6.6 Tying up the Proof 


We now return to the proof of the Local Correction Theorem, Theorem 5.4.3. It follows 
from Lemma 5.6.23 and Lemma 5.6.22 and Theorem 3.4.2. 


5.7 Implementing Local Checking in Real Networks 


In this chapter, we described a stabilizing snapshot and reset protocol for link sub- 
systems. This protocol was used to transform an automaton N that was locally cor- 
rectable to some predicate L intoa UIOA Nt that stabilizes to the behaviors of N(c)|L 
for some constant c. To make the snapshot/reset protocol stabilizing we numbered 
snapshot requests and responses; we also relied on the fact that each link was a UDL. In 
practice, however, there is an even simpler way of making the snapshot /reset protocol 


stabilizing. This can be done using timers. 


Suppose there is a known bound on the length of time a packet can be stored 
in a link and a known bound on the length of time between the delivery of a request 
packet and the sending of a matching response packet. Then by controlling the interval 
between successive snapshot/reset phases it is easily possible to obtain a stabilizing 
snapshot protocol. The interval is chosen to be large enough such that all packets from 
the previous phase will have disappeared at the end of the interval. This solution was 
advocated by us in [APV91a] and was implemented in a trial implementation on the 


Autonet [MAMT90]. 
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To keep our model simple, however, we have assumed that all lower bounds (on 
the time between events) are zero. Thus we have no way to model timers, which are 
needed to control the interval between phases. While we will not describe the timer 
based scheme formally, the reader interested in practical applications should be aware 
of the simplicity of a timer based scheme. Notice also that the maximum value of the 
timer need only be some small multiple of the worst-case round trip delay on a single 


link. We call such timers local timers. 


In most real networks, each node sends “keep-alive” packets periodically on every 
link in order to detect failures of adjacent links. If no keep-alive packet arrives before 
a local timer expires, the link is assumed to have failed. Thus, it is common practice 
to assume time bounds for the delivery and processing of packets. Note also that the 
snapshot and reset packets used for local checking can be “piggy-backed” on these 


keep-alive packets without any appreciable loss in efficiency. 


5.8 Summary 


The two main contributions of this chapter are the definition of Unit Storage Data 
Links in Section 5.1.2, and the notion of local checking and correction (Section 5.3 and 


Local Correction theorem, Theorem 5.4.3). 


In a stabilizing setting it is necessary to define Data Links that have bounded 
storage. First, such models correspond to physical reality. Second, they avoid the 
theoretical problems with unbounded storage links. We have chosen unit storage links 
(UDLs) because they are practical (see Section 5.2) and they can be modelled elegantly. 
We have also defined a stabilizing interface to a UDL. This is done by having the link 
periodically deliver a free signal (to avoid deadlock) and by having the sender keep a 
variable that indicates whether the link is free. We hope the UDL model will be used 
by others. 


Intuitively, a protocol is locally checkable if whenever the protocol is in a bad state, 
some pair of neighbors can detect this fact. Intuitively, a protocol is locally correctable 
if the protocol can be corrected to a good global state by independently correcting the 
states of each link subsystem. A link subsystem is just a pair of neighboring nodes 


and the links between them. 
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Local checkability is not a very new or surprising idea. For example, many authors 
have proposed local methods for detecting termination and deadlocks. In a stabilizing 
setting, the intuitive notion of local checkability was first referred to in a paper by 
[AK Y90]. However, their reference to this concept (during the description of a spanning 


tree protocol) was brief and intuitive. 


Our contribution has been to make precise the notion of local checkability, and 
to show that this is a useful and pervasive concept. Our definition (Definition 5.3.6) 
has some subtle aspects: for example, the requirement that each local predicate be 
closed may not be immediately obvious. Also, we have implemented local checking by 
doing a snapshot of each link subsystem. Now, a number of practical, self-stabilizing 
protocols (e.g., [Per85]) do what essentially amounts to local checking in the following 
way. Periodically each node sends its state to all its neighbors. However, (as we will 
show in Chapter 8) such periodic sending of state is not always sufficient to do local 
checking. Periodic sending is sufficient if all local predicates can be separated into 
what we call (see Chapter 8) one-way predicates. In the general case, a snapshot is 


required. 


The idea of local correction is more unusual. It is perhaps surprising that there 
are non-trivial protocols with this property. As we will see in Chapter 6, an easy way 
to ensure local correction is to first build a spanning tree of the network and then do 
local correction using the tree. Luckily, there is another class of protocols that are 
locally correctable: these are protocols that work in dynamic networks in which links 


can fail and recover. We will see an important example of such protocols in Chapter 


7. 


Local checking and correction is practical. In most real networks, each node sends 
“keep-alive” packets periodically on every link. The packets used for local checking 


and correction can be “piggy-backed” on these keep-alive packets. 
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Chapter 6 


Stabilizing Mutual Exclusion and 


Tree Correction 


The main result of Chapter 5 was the Local Correction theorem, Theorem 5.4.3. In this 
chapter we will consider a simple application of the Local Correction theorem to the 
problem of mutual exclusion. Section 6.1 gives an overview of our stabilizing mutual 
exclusion protocol. Section 6.2 formally defines our mutual exclusion protocol as a 
network automaton. Section 6.3 describes how local checking and correction is added 


to the automaton to create a stabilizing solution to the mutual exclusion solution. 


The last two sections of this chapter extract some general principles from the 
example of stabilizing mutual exclusion. In Section 6.4, we discuss a weaker notion of 
local checkability called weak local checkability. We show that in certain cases, a simple 
heuristic of removing unexpected packet transitions can be used to transform a protocol 
that is weakly locally checkable into a locally checkable protocol. This heuristic has 


proved to be quite useful, and is used in later chapters. 


Finally, in Section 6.5, we show (informally) an important result, the Tree Cor- 
rection theorem. This theorem states that any locally checkable protocol on a tree 
topology can be efficiently stabilized. In other words, if the underlying topology is a 
tree we can dispense with the need for the (stronger) local correctability condition. 
This theorem is the counterpart to a similar theorem proved in Chapter 4 for shared 


memory systems. 
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6.1 Overview of Token Passing Protocol 


Token passing is one way of ensuring mutual exclusion in a network. If there is only 
one token in a network, a node can go into the critical region when it receives a token. 
In the I/O automaton model this is typically modelled by adding an output action to 
every node by which the node can give permission to some external user to go into 
the critical section, and adding an input action that tells the node when the user has 
finished with the critical section. For simplicity we ignore this extra piece of modelling 


which is important for providing a modular interface to other subsystems. 


Token protocols that stabilize in time proportional to the height of the tree have 
been described before by [DolevIM90] in a shared memory setting. However, we believe 
our protocol is simpler and more transparent. While the mutual exclusion protocol is 
very simple, its simplicity makes it a good candidate to understand how the general 


method of local correction developed in Chapter 5 can be applied. 
Stabilizing spanning tree protocols have been well studied [AK Y90],/AG90],|AV91]. 


Thus we can reduce the problem of token passing on an arbitrary connected network 
to that of token passing on a tree. Instead of modelling the interface to the network 
automaton that computes the tree, we will (for simplicity) assume that the network 
graph is a tree. Thus we are dealing with a network automaton of the form N = 


Net(G, N) where G is a tree graph. Formally: 


Definition 6.1.1 A tree graph T is a topology graph such that: 


e T is a rooted tree. (i.e, there is a distinguished node called the root and there is 


a unique path between any node and the root.) 
e For every edge (u,v) in G, the leader function I(u,v) = u if u is the parent of v 


in the tree, and I(u,v) =v if v is the parent of u in the tree. 


Definition 6.1.2 A tree automaton is a network automaton whose topology graph is 


a tree graph. 


For a tree graph we will refer to the leader function | as parent for obvious reasons. 


The simplest mutual exclusion algorithm is to pass a token along some tour (e.g., 


DFS) of the tree; a node can go “critical” when it has the token. But such a protocol 
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Figure 6.1: Adding a pointer to each node makes token passing on a tree locally checkable. 


is not locally checkable; if there are two tokens at two links, each link subsystem may 


not locally detect a problem. 


Once we see this, it is easy to add a small amount of state to make the token protocol 
locally checkable. We add a pointer, pointer, to each node z such that pointer, points 
to where the token is in the tree; also if the token is at node u, then pointer, = nil. 


This is shown in Figure 6.1. 


6.2 Specification of Token Passing Protocol 


Our stabilizing token passing protocol is based on this idea and is specified by VN = 
Net(T, N) where: 


e T is a tree graph. 


e Each u € V is a UIOA of the form described in in Figure 6.2. 


We assume that there is a function neighborSet(u) that lists the set of neighboring 
nodes of node wu. (Recall that we allowed node automata to depend on the set of 
neighboring nodes and the leader function.) We assume there is some fixed circular 
ordering on neighborSet(u) and that there is a function Succ, which when given a 
neighbor v as its argument produces the next neighbor in the ordering. The only 
packet sent by this automaton is a token packet token € Pigig. We use the following 


additional variables: 
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e pointer, € neighborSet(u) U nil (*pointer to token*) 
e last, € neighborSet(u) (*last neighbor u received token from*) 
e queue,|v] (*queue that either contains exactly one token packet or is empty.*) 


e free,|v| : boolean (*true if no packet in transit on link to v *). 


Note that we can use domain restriction to ensure that the outbound queue for a 
link either contains a token or is empty. This can be done by using a single value to 
encode the queue that is either token or nil. We use a queue to keep the definition of 
N, compatible with the definition of a node automaton in Chapter 5. To make the 
token passing automaton a node automaton, we also have to pass the token by first 
enqueueing the token in the outbound queue for a link. Then, a second separate action 
is required to send the token from the queue to the link. However, by making N, a 


node automaton, we can apply the Local Correction Theorem of Chapter 5. 


Notice the code we have given does not specify start states because each N, is a 


UIOA . 
Lemma 6.2.1 N = {N,,u € T} is a set of node automata for T with Pista = {token}. 


. Proof: Simple checking of the definitions given earlier. J 


6.3 Adding Local Checking and Correction 


To add local checking and correction, we need to define a link predicate set £ for 
N = Net(T,N). And to define £ we need to define a link predicate, L,,, for each 
edge (u,v) in T. Let havetoken(u,v) be the boolean condition that is true iff there 
is a token in either queue,|v], Quy», queue,(u], or Qvu. (Informally, havetoken(u,v) is 
true iff there is a token stored on any of the links between u and v or on the outbound 


queues for such links.) 


Intuitively, L,, should describe the legal states of the (u,v) subsystem when WN is 
behaving properly and there is exactly one token in the system. Consider such a good 
state and suppose the pointer at u is pointing to v. Then either the token is in transit 


between u and v OR the pointer at v is pointing away from w. 
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ENQUEUE,,y (*output action to enqueue token in queue for neighbor v*) 


Preconditions: 
pointer, = nil; (* u has token?*) 
v = Succ,(last,); (* v is the next neighbor after last ?*) 
Effect: 
pointer, :=v (* point towards where token is sent*) 
Add token to queue,,|v| (*store token in outbound queue*) 
SEND,y,»( token) (*output action to send token to neighbor v*) 
Preconditions: 
free, |v] = true (* link to v free?*) 
token is head of queue,,|v] 
Effect: 
free, |v] = false (* set to false until link says its free*) 
Remove token from head of queue,,|v] (*empty outbound queue*) 
FREEu,» (*link to v says it is free, input action*) 
Effect: 


free, |v] = true 


RECEIVEy,,,( token) (*input action, token received from neighbor v*) 
Effect: 

If pointer, = v then (* token received on expected link?*) 

pointer, := nil (* accept token*) 

last, := v (* update last*) 


All actions are in a separate class with upper bound ft,. 


Figure 6.2: Actions for a node u with respect to a neighbor v in the stabilizing token passing protocol. 
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Definition 6.3.1 Local Predicates for Mutual Exclusion: We define the local 
predicate Ly of N to hold iff all the following conditions hold in the (u,v) subsystem: 


e Exactly one of (pointer, # v) or havetoken(u,v) or (pointer, # wu) is true. 


e There can be at most one token packet in the combination of queue,|v], Quy, 


queue, |u] and Qyu- 


Let L be the link predicate set that consists of Lu,» for each edge (u,v) in T. Let L 
denote Conj(L). 


Then we can show that: 


Lemma 6.3.2 The network automaton N = Net(T, N) is locally checkable for L using 
link predicate set L. 


Proof: By definition L = Conj(L). We also need to show that each L,,, is a closed 
predicate. Recall that we say that a state s of NV satisfies L,,» iff (s|u, s|(u,v), 5|(v,u), s|v) € 
Lu». Thus we need to show that for any transition (s,7,5) of N, if s satisfies Lu», 


then so does &. So assume that s satisfies L,,y. 


Clearly we need only consider actions at nodes u and v since only such actions can 
affect variables of the (u,v) subsystem. Also we need only consider actions at node u, 


because the argument for actions at node v is symmetrical. 


Suppose 7 is a ENQUEUE,,, event. Then in s, pointer, = nil. Thus we can infer that 
in s, havetoken(u,v) = false and pointer, = u. If e = v, then in 3, havetoken(u,v) = 
true and pointer, = v and pointer, = u. Also in & there is no token in Qu», queue, |u| 
and Qyu- On the other hand, if x 4 v, then in 5, havetoken(u,v) = false and pointer, # 


v and pointer, =u. In either case, § satisfies L,,. 


Suppose 7 is a SEND, (token) event. If « # v, then this action does not affect any 
of the concerned variables. So suppose x = v. Then in s, token is the head of queue, |v]. 
Thus we can infer that in s, pointer, = x, havetoken(u,v) = true, and pointer, = u. 
Also in s there is no token in Qu», queue,|u] and Q,,4. All this action does is to move 


the token from queue, |v] to Qu». Thus 4 satisfies Ly». 
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Empty queue,|v] (*remove any token stored in queue™*) 


If u is the parent of v then 


If pointer, = v then pointer, = nil (* take away token from child subtree*) 
Else (*v is parent of u*) 
pointer, = v (* point towards parent*) 


Figure 6.3: Code for f(s|u, v), the reset function applied at node u with respect to neighbor v 


Suppose 7 is a FREE, , event. Then this action does not change the concerned 


variables and hence § satisfies L,,y. 


Suppose 7 is a RECEIVE,,,(token) event. Then in s, Qy. = token. Thus we can 
infer that in s, pointer, = v, havetoken(u,v) = true and pointer, = u. Also in s 
there is no token in Qu, queue,|u] and queue,|v]. Thus in 3, the only change is that 


Qvu = nil, havetoken(u,v) = false, and pointer, = nil. 


The most interesting case is if 7 is a RECEIVE,,,(token) event with « #4 v. We 
consider two cases. Suppose in s, pointer, # «. Then, since the code will not accept the 
token in this case, it is easy to see that in 8, the values of pointer, pointer, Quy», Qvu, 
queue, |u] and queue,,|v| are identical to their values in s. Hence § satisfies L,,,,. Suppose 
in s, pointer, = z. Then we can infer that in s, pointer, = u and havetoken(u,v) = 
false. Also in 8, pointer, = nil and pointer, = u and havetoken(u,v) = false. Thus 
satisfies L,,. I 


The next thing we have to do is to specify a local reset function f to correct each 
(u,v) subsystem. Our idea is very simple. Let us define a partial order on pairs of 
neighbors in T such that “any edge e is less than any edge below e in T”. More 
precisely, we let {u,v} ~< {v,w} iff u is the parent of v and v is the parent of w. Also, 
{u,v} A {w,zx} if the two pairs do not have a node in common. We let < be the 


transitive closure of <~. Thus, the partial order directly reflects the structure of T. 


To allow f to be a reset function using < we must ensure that applying f to the 
state of v with respect to child w does not affect the stability of L,,. This can be 
achieved by the following reset function f described in Figure 6.3. 
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Lemma 6.3.3 The function f defined in Fig. 6.3 1s a local reset function for network 
automaton N = Net(T,N) with respect to link predicate set £L and partial order <. 


Proof: Consider any edge (u,v) and suppose that u is the parent of v; the reverse 


case is symmetrical. We check the two conditions in Definition 5.3.7. 


e Correction: We need to show that (f(s|u,v), nil, nil, f(s|vu,u)) € Lu». Consider 
f(s|u,v). From the code in Figure 6.3 we see that in this node state, pointer, # 
v and queue,|v| is empty. Consider f(s|v,u). From the code in Figure 6.3 


we see that in this node state, pointer, = u and queue,|u] is empty. Thus 


(f(slu,v), nil, nal, f(s|v,u)) € Lup. 


e Stability: We only need to check the stability condition for links less than 
(u,v) in the partial order. Since v is the child of u it suffices to show the 
following: for any neighbor w of v, if (s|u,s|(u,v),s|(v,u),s|v) € Lu» then 
(s|u, s|(w,v), s|(v, uw), f(s|v,w)) € Lu». But if we change the state of node v 
from s|v to f(s|v,w), queue,|u] remains unchanged. Next, if pointer, # u in s|v, 
then pointer, # win f(s|v,w). Similarly, if pointer, = u in s|v, then pointer, = u 


in f(s|v,w). Taken together, these facts imply the stability condition. 


Let height(T) denote the height of tree T which in turn is the length of the longest 
path from the root to a leaf. Clearly height(f) = height(T). 


The next lemma that follows directly from the previous lemmas and the definitions: 


Lemma 6.3.4 The network automaton N = Net(T,N) is locally correctable to L 
using link predicate set L and the reset function f defined in Fig. 6.8. 


The following theorem follows directly by applying the Local Correction theorem 
(Theorem 5.4.3). 


Theorem 6.3.5 Let N be the set of node automata defined in Figure 6.2, L be the 
local predicate set defined in Definition 6.3.1, and f be the local reset function defined 
in Figure 6.8. Let N* = Augment(N,L,f), Then N* stabilizes to the behaviors of 
N(ty)|L in time tq -height(T), where ty and ty are constants. 
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Let the symbol * denote any node in the tree. We define the “correct” set of 
behaviors of a token passing system using a set B. Informally, B is the set of behaviors 
consisting of phases in which a token is received at a node, then enqueued at the same 
node, and then passed to a neighbor of the node. Also, we require that a token will 


be received periodically at every node. 


Definition 6.3.6 We define the set B of correct behaviors of a token passing system 
as follows. B is the set containing any behavior 3 that only has actions of N and such 
that: 


e For any u, after any RECEIVE..,,(token) event in GB the neat event other than a 
FREE, event 1s an ENQUEUE,,,. event. 


e For any u,v, after any ENQUEUE,,,(token) event in 8 the nezt event other than 
a FREE,,,. event is a SEND,,,(token) event. 


e For any u,v, after any SEND,,»(token) event in 8B the neat event other than a 


FREE, event is a RECEIVE,,(token) event. 


e For any u, and any suffiz y of 8, a RECEIVE,.,,(token) will occur in c-n time 


after the start of y, where c is some constant and n is the number of nodes in T. 


We first argue informally that the behaviors of M(t,)|L are in B. 


To make a verbal argument, we introduce some intuitive terminology. Let us say 
that a token is in transit between nodes u and v if havetoken(u,v) = true. We say 


that a token is at node u if pointer, = nil. 


We first see that for any s € L, there is at least one token in s. An intuitive 
explanation for this is as follows. We will define a search procedure to find a token 
in state s. Start at any node uw in the tree. If pointer, = nil then there is a token 
at u. If pointer, = v then from the local predicates, either there is a token in transit 
between u and v or pointer, # u. If pointer, 4 u we continue the search procedure 
recursively at node v. Since we never backtrack, the search procedure cannot continue 
indefinitely without encountering some leaf node w such that pointer,, 4 x, where z 
is the parent of w. But if w is a leaf node and pointer,, # x then, pointer,, = nil and 


hence the token is at w. 
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Thus we know that there is at least one token in state s. Suppose this token is at 
node u. Then by induction on the length of the path between u and any node w  u, 
it is easy to see that pointer,, # nil. Similarly for any edge (w,z), by induction on the 
length of the path between u and (w,z) we can show that the token is not in transit 
between w and zx. A similar argument shows that if in s the token is in transit between 
u and v, then the token is not at any node z, nor is it in transit on any other edge. 


(w,az). Hence there is exactly one token in state s. 


Once we know that there is exactly one token in any state s of N(t,)|L, it is quite 


easy to prove that the behavior corresponding to any execution a of N(t,)|L is in B. 


For example consider any execution a and any state s; in a immediately following a 
RECEIVE,.,(token) event for some node u. Then it is easy to see that in s;, pointer, = 
nil. This predicate will continue to hold until a an ENQUEUE,, event occurs. But if 
pointer, = nilin a state s, then (since there can be only one token in state s) the only 
actions enabled are ENQUEUE,,,. or a FREE, event. Similar arguments can be used to 
show the behavior of a satisfies the next two properties that characterize a behavior 
in B However, we also need to show the fourth property of a behavior: that for any 
u, and any suffix y of a, a RECEIVE,.,(token) will occur in c-n time after the start of 
yy, where c is some constant. This can be shown by an inductive argument using the 


properties of the Succ function. 


Thus we can show: 


Theorem 6.3.7 Let N* be the automaton defined in Theorem 6.3.5. Then Nt stabi- 


lizes to the behaviors in problem B in time t,-height(T), where ty and t, are constants. 


6.4 Removing Unexpected Packet Transitions 


Consider the following modification of the token passing protocol described in 
Figure 6.2. The modification is shown in Figure 6.4. The only difference is that the 
routine to receive a token at a node u from node v has been changed. The only change 
is that we no longer check whether pointer,, = v before accepting the token. Let us 


call the resulting tree automaton N*. 


Assume, however, that we continue to use the definitions of £, L, and L,,, from 


Definition 6.3.1. Then it is quite easy to prove that N* 1s not locally checkable with 
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MODIFIEDRECEIVEy,,(token) (*token is received from neighbor v*) 
Effect: 

pointer, := nil (* accept token*) 

last, :=v (* update last*) 


All external actions are in a separate class with upper bound ¢,. 


Figure 6.4: Modified Code for a node u in a token passing protocol. The remaining code is identical to 
the code in Figure 6.2 


respect to L and £L. This follows from the fact that L,, is not a closed predicate of 
N*. 

Despite this it is not hard to prove that the behaviors of N*|L are exactly the 
behaviors of N|L. In fact, if we are allowed the luxury of specifying initial states, 
N*|L is a “natural” IOA to solve the token passing problem. Suppose a protocol 
designer has started with M*|LZ and now wishes to construct a UIOA that stabilizes 
to the behaviors specified by the token passing problem. It is interesting to note that 


this can be done by the following two step process: 


e Transform N* into N by adding the extra check shown in Figure 6.2. 


e Transform NV into N* as shown earlier. 


We would like to abstract this process. First, note that the extra check added in 
going from A* into N essentially amounts to the following heuristic: if we receive 
an “unexpected” packet p at node u from node v, then we do not process p. Notice 
that in NV, a token received from v when pointer, # v (see Figure 6.5) is certainly 
“unexpected”. We will formalize this intuitive notion of an “unexpected” packet below. 
However, for the present it is important to intuitively understand why such checks for 
unexpected packets are useful. Consider a transition (s,7,5) of NV such that s satisfies 
Ly, but not Ly. Then it is quite possible that there is an “unexpected” packet on 
channel Q,,,. in state s. By adding checks for such packets in NV, we can ensure that 


8 satisfies L,,. Also note that these checks do not affect the “correct” executions of 


NIL. 
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Token receipt 


ointer 
Pp u 


Figure 6.5: Receiving a token on a link that is not being pointed to is an unexpected packet transition. In 
general, an unexpected packet transition is a packet reception that could never have occurred if the receiving 
link subsystem was in a good state. 


We now formalize these observations. First, we define the notion of weak local 
checkability. Intuitively, we remove the requirement that that each LD, be a closed 


predicate and instead require only that ZL is a closed predicate. 


Definition 6.4.1 A network automaton N is weakly checkable for predicate L using 
link predicate set L tf: 


e £ is a link predicate set for N and L > Conj(L). 


e For any transition (s,7,5) of N, ifs © L thense L 


Ideally, we would like to prove that any weakly checkable protocol can be trans- 
formed into an “equivalent” locally checkable protocol. While we do not know how 
to do this in general, we can can obtain such a result if the automaton is also locally 
extensible. Intuitively, an automaton is locally extensible with respect to a link predi- 
cate set if any pair of “good” adjacent link subsystem states can be extended to form 


a “good” state of the entire automaton. 


Definition 6.4.2 A network automaton N = Net(G,N) is locally extensible with 
respect to link predicate set L= {Lu} if the following condition is true: 

For any two adjacent edges (u,v) and (v,w) in G, if x € Lu, and y © Ly then 
there is some state s € Conj(L) such that x = (s|u,s|(u,v),s|(v,u),s|v) and y = 


(s|v, s|(v,w), 8|(w,v), s/w). 


To transform a locally extensible and weakly checkable protocol M* into an “equiv- 
alent” locally checkable protocol, we will add checks to N* for “unexpected” packets. 


We formalize this notion of an “unexpected packet” (see Figure 6.5) as follows: 
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Definition 6.4.3 Consider a network automaton N with link predicate set L = {Lu} 
and some pair of neighbors u,v in N. We say that a transition (s, 7,5) is an unexpected 


packet transition at u with respect to v if: 


mw is a RECEIVE, u(p) event and there is no a,b such that (s|u,a,p,6) € Lup». 


For example, in Figure 6.2, a (s, RECEIVE,,,(token), ) transition with s.pointer, # 
v is an unexpected packet transition at u with respect to v. We can now state a simple 


theorem. 


Theorem 6.4.4 Consider a network automaton N* = Net(G, N) that is weakly check- 
able for predicate L with respect to predicate set L. Suppose further that N is locally 
extensible with respect to L. Then it is possible to construct another automaton N 
such that: 


e N is an automaton for graph G and the executions of N|L are identical to the 
executions of N*|L 


e N is locally checkable for predicate L with respect to predicate set L. 


Proof: We transform \* into N by replacing all unexpected packet transitions (s, 7, 5) 
in N* by the null transition (s,7,5). Then we use the local extensibility property to 
show each L,,,, is a closed local predicate of VN. Il 


6.5 Tree Correction Theorem 


The reader who has read Chapter 4 might suspect that any locally checkable protocol 
on trees can be made locally correctable. Thus for tree automata it seems plausible that 
we do not need the stronger hypothesis that the tree automaton be locally correctable. 
This is indeed true. Compare the following theorem to Theorem 4.3.1 (but be aware 
that Theorem 4.3.1 applies to shared memory tree automata) and the Local Correction 
theorem (Theorem 5.4.3). 


Theorem 6.5.1 Tree Correction: Consider any tree automaton T = Net(T,N) 
that is locally checkable for L using link predicate set L. Then there exists some Nt 
that is a UIOA for graph G and constants c and é such that N* stabilizes to the 
behaviors of N(c)|L in time é- height(T). 
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The proof of this theorem is extremely similar to that of Theorem 4.3.1 of Chapter 
4. However, since we can no longer do snapshots and resets atomically, we need to 
use the local snapshot and reset protocols defined in Chapter 5. Thus we have to 
add actions for local checking and correction as in the proof of the Local Correction 
Theorem, Theorem 5.4.3. However, there are two differences. We will assume that 
for every (u,v) subsystem in which v is the parent of u, the child u performs the 
checking/correction. The local snapshot protocol is identical to the protocol of Chapter 
5, but the local reset protocol is a little different. This is sketched in Figure 6.6. The 
figure should be compared with the right hand diagram in Figure 5.9. 


The basic idea is that the reset response carries the state b of the parent at the 
instant the response was sent. When the child gets the response, the child sets its 
state to f(b), where f is a function that we describe next. Basically, f is chosen such 
that for every state b of the parent node v, (f(b), nil, nil, b) € Lu. In other words, f 
is chosen so that we can reset the link subsystem to a good state by only changing 
the state of the child node. Of course, that is the basic idea in the proof of Theorem 
4.3.1. The only tricky part is to argue that we can find such a function f. The proof 
is again similar to the proof of Theorem 4.3.1: we first normalize the original protocol 
T to get rid of “useless” node states that can never occur in global states in which 
all local predicates are true. We then show that the required function f exists for the 


normalized protocol. 


The Tree Correction theorem is not a corollary of the Local Correction theorem. 
Recall in Chapter 5 that when we applied a local reset function to the state of leader 
node u with respect to node v, the resulting state of node u can only depend on the 
previous state of node u. However, in the proof of the Tree Correction theorem we 


require the the resulting state of node u to depend on the previous state of node v! 


Finally, we note that we could have derived the stabilizing mutual exclusion pro- 
tocol by showing that it was locally checkable and then using the Tree Correction 


theorem directly. 


6.6 Summary 


In Chapter 4, we showed (in a shared memory model) that any locally checkable 


protocol on a tree could be stabilized in time propostional to the height of the tree. 
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Figure 6.6: Sketch of a reset phase used for Tree Correction. Node v is the parent of node u in the tree. 


This chapter shows that this theorem can be extended to the message passing model 
(Tree Correction theorem, Theorem 6.5.1). We also described a simple application — 
the problem of token passing on a tree. A stabilizing solution to this problem can be 


derived using either the Local Correction or Tree Correction theorems. 


The token passing problem suggests a simple strategy for stabilizing protocols. 
First, we try to add a small amount of state to make the protocol locally checkable. 
Recall that we added a pointer to each node for this purpose in the token passing 
protocol. Then we combine the original protocol with another protocol that computes a 
spanning tree. Finally, we do local correction on the resulting spanning tree. Although 
we have not done so, it is important to formally describe how an arbitrary protocol P 
could be composed with a spanning tree protocol so that the net effect is the same as 


if P were running on the final tree. 


Local checkability requires that each local predicate be a closed predicate. Re- 
moving unexpected packet transitions (Figure 6.5) is a useful heuristic that is often 


sufficient to ensure that each local predicate is closed. 


The token passing protocol on a tree can be generalized to passing a constant 
number of tokens on a tree. In this case, we replace the pointer variable pointer,, at 
each node u by a variable token_count,|v| (one for each neighbor v) that keeps track 


of the number of tokens that are in the direction of neighbor v. The local predicates 
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have also to be suitably modified. 
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Chapter 7 


Stabilizing Network Reset 


The stabilizing network reset protocol described in this chapter is the link between the 
previous two chapters (which were about Local Correction) and the last two chapters 
of the thesis (which are about Global Correction). 


The major service provided by a network reset protocol is synchronization of the 
nodes in a network. In the first section of this chapter, we informally introduce the 
concept of synchronization, and discuss why this service is useful. In Section 7.2 we 
review existing reset protocols. Then in Section 7.3 we specify the reset problem. 
Previous specifications of reset protocols have been state-based. Our specification 
of the reset problem is novel in that it is based on external behaviors. In the next 
section (Section 7.4), we give an overview of our reset protocol. Then in Section 7.5, 


we formally specify our reset protocol using a reset automaton. 


Sections 7.6 and 7.7 are devoted to showing that the reset automaton is a stabilizing 
solution to the reset problem. We do this using the Local Correction theorem developed 
in Chapter 5. We show that the reset automaton is locally checkable by exhibiting a 
set of closed local predicates for the reset automaton. Then we show that the reset 
automaton is locally correctable by showing that the local predicates are independent 
—i.e., the partial order that formalizes the dependency relation between the predicates 
is the trivial partial order. Thus the reset protocol stabilizes in constant time. Next, 
(in Section 7.7), we show that once the reset automaton is in a state that satisfies 
all local predicates, then the behaviors exhibited by the reset automaton are indeed 
the behaviors specified by the reset problem. This completes the proof that the reset 


automaton is a stabilizing solution to the reset problem. 
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The last two sections of the chapter try to abstract some general principles based 
on the example of the stabilizing reset protocol. The reset protocol in this chapter is 
based on an existing non-stabilizing reset protocol [AAG87] that works in networks 
where links can fail and recover. Section 7.8 suggests that this is no accident — locally 
checkable protocols that work in networks where links can fail and recover are likely 


to also be locally correctable. 


Because this chapter is very long, we offer two suggestions for reading. First, 
constantly consult the roadmaps at the beginning of the chapter and each long section. 
Second, it is hard to appreciate the specification of the reset problem until one sees why 
it is useful. Chapters 8 and 9 describe important applications of the reset protocol. 
Readers may prefer to read Chapter 8 after reading the specification in Section 7.3 
and before reading the rest of Chapter 7. 


7.1 Synchronization 


A reset protocol is used to synchronize all the nodes in a network. Before we describe 
what we mean by synchronizing all the nodes in a network, it is helpful to understand a 
form of synchronization between a pair of nodes in a network. Such synchronization is 
provided by a Data Link protocol [Spi88a]. We will then see that in essence a network 
reset protocol generalizes the guarantees of a Data Link protocol to multiple nodes in 


a network. 


7.1.1 Data Link Synchronization 


Consider a pair of neighboring nodes in a network, say u and v. Suppose the physical 
link between two nodes in a network can periodically crash and recover. The Data 
Link protocol is responsible for providing notification (as to whether the link is up or 
down) to the users at u and v. Thus at each node, the Data link protocol issues Link 
up and Link down actions to signify that the link is operational or not operational 
respectively. If the network is asynchronous, it is impossible to provide a Link down 
event (or Link up event) at exactly the same instant at both u and v. But if Link up 
and down events are reported independently (and possibly at different times) at both 


ends, the Data Link must provide some additional functions to synchronize u and v. 
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The synchronization requirements for a Data Link protocol can be stated elegantly 
[Spi88a] as follows. First, for node u (or node v) we can define an operating interval at 
node u (or at node v) to be the time from a Link up event at that node until the next 
Link down event at that node. If an operating interval does not end with a Link down 
event we say that the interval is a final interval. Thus each execution of the Data Link 
protocol induces a set of operating intervals at both nodes. Then for synchronization, 
we require that there is a symmetric relation between intervals at the two nodes (called 
a mating relation in [AE86]) such that: 


e An operating interval can be mated to at most one other operating interval. 


e Suppose an operating interval a at u is mated to an operating interval § at 
v. Then the sequence of messages received by v in @ must be a prefix of the 
sequence of messages sent by wu in a. (This is often called the prefix property of 
Data Link protocols.) Also, if a is a final interval, then so is 6 and in this case 
the sequence of messages received by v in § must be identical to the sequence of 


messages sent by u in a. 


e Suppose an operating interval a at u is mated to an operating interval 6 at v 
and an operating interval a’ at u is mated to an operating interval @’ at v. Then 


if a’ occurs later than a then (’ occurs later than (. 


The mating relation for two nodes u and v is sketched using the time-space diagram 
shown in Figure 7.1. We have depicted Link up events by horizontal lines. The Link 
down events between Link down events are depicted by crosses. An arrow from v to u 
depicts a packet sent by v that is successfully delivered at wu, and vice versa. An arrow 
from v that does not reach u is a packet sent by v that is not delivered at u. In the 
figure, the second operating interval at u mates to the second operating interval at v. 
Also, the third operating interval at u mates to the fourth operating interval at v and 
the two intervals are final intervals. Notice that the sequence of packets received by 
u in its second interval is a prefix of the sequence of packets sent by v in its second 


interval. Also notice that all packets sent in the two final intervals are delivered. 


Why does this mating relation provide a useful form of synchronization? It does so 
because the synchronization relation guarantees that the behavior of a node during an 


operating interval is what might have occurred in some asynchronous execution of the 
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Figure 7.1: Illustrating the Mating Relation 


Data Link protocol in which there were no link failures. This is a crucial abstraction 


that allows users of the Data Link protocol to deal with failures in a simple way. 


Almost identical forms of synchronization are provided by virtual circuit protocols 
([Spi88a]) and transport protocols. Thus all these protocols essentially synchronize 


two nodes in a network. 


7.1.2 Network Synchronization 


The synchronization provided by a network reset protocol is a generalization of the 
synchronization guarantees of a Data link. A reset protocol synchronizes all the nodes 
in the network. Informally, the reset problem [Fin79] is to design a reset service that 
can be superimposed on any other distributed protocol P. The reset may be invoked 
at any node, and its effect is to output a signal at all the nodes of the system in 
a consistent way. We use X-messages to denote the messages sent and received by 


protocol P. 


By consistent, we mean the following. As in the Data Link protocol, the reset 
protocol induces signal intervals at each node (i.e., intervals between consecutive signal 
events at a node). Then for each pair of neighbors u and v we require that the signal 
intervals at u and v can be mated as described earlier. For example, if we replaced the 


Link up events with Signal events in Figure 7.1 and ignored Link down events, then 
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Figure 7.1 could equally well describe the pairwise mating relation provided by a reset 
protocol. Notice, however, that in a Data Link the Link up events occur independently 
on each link and so the the operating intervals on each link adjacent to a node u can 
be different. However, in a Reset protocol, the Signal events induce signal intervals on 


a per node basis. 


For example this means that the sequence of -messages sent by any node wu to 
any neighbor v after the last signal at wu must be equal to the sequence of /-messages 


received by v after its last signal. 


Why is this called a reset service? Suppose S,, defines the set of “legal” start states 
for every node u. Suppose the “legal executions” of P are those executions in which 
the initial state of every node u belongs to S$, and the initial state of every channel 
is a state in which the channel contains no %-messages. It’s natural to define the 
legal states of P as those states that occur in legal executions of P. Suppose we now 
superimpose a reset service over P. Suppose also that whenever a node uw receives a 
signal, we locally reset the state of node uw (i.e., the local state of protocol P at node 
u) to some state in S,. Then after every node has received its last signal, protocol P 
is in a legal state. In other words, the signals provide a consistent time point for every 
node u such that we can globally reset protocol P to a legal state by locally resetting 
each node wu at its time point. The time point for each node is the instant it receives 


a signal. 


Thus a global reset service is much like the reset button on a computer. After the 
reset button is pushed, the computer is restored to a “good” state. However, globally 
resetting a network to a “good” state is much more challenging than resetting a single 
node. In a network, reset requests can arrive at any node and the signals must be 
delivered at every node in a consistent fashion. Ideally, we would like the signals to be 
delivered to every node at the same instant. However, since this seems to be impossible 
in a distributed system, we settle for delivering the signal in a consistent manner (see 


above). 


Why is a reset service useful? It was introduced in [Gal76, Fin79] as a tool for 
converting any protocol that works in a so-called static network to work in a so-called 
dynamic network. A static network, as the name suggests, is a network in which 
the topology of the network remains fixed during the execution of the algorithm. A 
dynamic network, by contrast, is a network in which nodes and links can crash, thereby 


changing the topology. However, it is assumed that topology changes eventually stop 
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and that some node in the final topology can detect the fact that there has been 
a topology change. Roughly, the idea behind [Fin79] is that any node that detects 
a topology change makes a reset request. If successful, the reset request results in 
restarting the static protocol. This methodology is quite practical. For instance, the 


Autonet local area network uses a version of [Fin79] to cope with failures. 


Besides its use in dynamic networks, a reset protocol is also useful in a stabilizing 
setting. As we show in Chapters 8 and 9, a stabilizing reset protocol is an important tool 
for the design of other stabilizing protocols. Notice that in order to use the reset service 
in dynamic networks |Fin79], some node must detect the last topological change. More 
generally, suppose that any bad state of a network protocol can be detected locally by 
some node. This corresponds to what we have called a locally checkable protocol in 
Chapter 5. As we will see in Chapter 8, any locally checkable protocol can be stabilized 
using a stabilizing reset protocol. Intuitively, our idea is similar to that of |[Fin79]. We 
add actions to each node to make a reset request if the node locally detects a bad state 


of the network. 


The technique of using a reset protocol to stabilize a locally checkable protocol 
is quite different from the techniques developed in Chapter 5 and 6. In Chapter 5, 
a locally checkable and correctable protocol is stabilized by doing independent local 
resets of each link subsystem. In Chapter 6, a locally checkable protocol on a tree is 


also stabilized by doing independent local resets of each link subsystem. 


By contrast, we can stabilize a locally checkable (but perhaps not locally cor- 
rectable) protocol by doing a coordinated global reset of the entire network. As one 
might guess, there is a performance penalty in using a network reset. The stabilization 
time of a solution that uses local correction is proportional to the height of the under- 
lying partial order (see Local Correction Theorem, Theorem 5.4.3). The stabilization 
time of a solution that uses tree correction is proportional to the height of the tree 


(see Tree Correction Theorem, Theorem 6.5.1). 


However, the stabilization time of a solution that uses global correction is propor- 
tional to the number of nodes in the network. Because the height of the partial order 
(or the height of a tree) typically is smaller than the number of nodes, we have the 
following rule of thumb. If a protocol is locally correctable or runs on a tree, then it 
pays to use the techniques of Chapter 5 or 6. However, if a locally checkable protocol 
cannot be shown to be locally correctable, then the network reset approach provides 


a (perhaps less efficient) stabilizing solution. It is perhaps elegant that the network 
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reset protocol is itself stabilized using the local reset approach of Chapter 5. 


In this chapter we introduce the most efficient known stabilizing network reset 
protocol. We do so by applying the method of local checking and correction of Chapter 
5 to an existing reset protocol described in [A AG87] The space overhead of the protocol 


is logarithmic. Our reset protocol stabilizes in constant time. 


7.2 Existing Solutions 


In Chapter 4, we described a stabilizing reset protocol due to Arora and Gouda [AG90}. 
Their protocol was described in terms of a shared memory model but it appears that 
it can be adapted to work in our message passing model. [AG90] also describes a 
stabilizing protocol to build a spanning tree of the network. For the spanning tree 
protocol, it is assumed that processes have unique identifiers, and that there is some 
a priort bound K on the number of nodes in the network. The IDs and K cannot be 
corrupted by transient errors. The stabilization time of the spanning tree protocol is 
O(K), where K is a non-volatile bound on the number of nodes in the network. Their 
protocol will also work correctly if K is an upper bound on the diameter of the final 
network. However, in a network in which the topology can change arbitrarily, often 
the only reasonable bound on the diameter of the network is a bound on the number 
of nodes in the network. Secondly, as we will see in Chapter 8, their spanning tree 


protocol is based on a routing algorithm that works poorly in practice. 


Katz and Perry [KP90] describe a stabilizing reset protocol. Their approach re- 
quires the election of a leader and the stabilization time of their approach is O(n?) 


where n is the number of nodes in the network. 


On the other hand, the reset protocol we describe does not require node IDs, and 
takes O(n) time to stabilize, where n is the actual number of nodes in the network. 
Our protocol does not require the computation of a spanning tree. Thus it can be 


used to build a stabilizing spanning tree protocol as we show in the Chapter 8. 
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7.3 Specifying the Desired Behaviors of a Reset 


Protocol 


This section is divided into four subsections. First, we describe the external interface 
to the reset protocol. Then, we give an overview of the specification and some of 
its difficulties. After this motivation, we formally specify the reset problem and then 


briefly discuss alternative specifications. 


7.3.1 Interface to Reset Protocol 


We first describe the external interface to the reset protocol. 


A reset service is modelled as a network automaton R = Net(G,N). The external 
interface for any node wu in the network is shown in Figure 7.2. We have the usual 
interfaces to send and receive packets between neighbors as described in Chapter 5. 
However, in addition each node also has interfaces to send and receive messages on 
behalf of external users of the reset service. We assume that every message m is drawn 
from some message alphabet %, and that © C Puaata, where Piata is the data packet 
alphabet used by the network automaton. Intuitively, the messages sent by users to 


the reset service will be relayed between nodes of the network using packets. 


Thus node w also has an input action SENDM,,,(m) by which an external user can 
send a message to neighbor v. Similarly node u has an output action RECEIVEM,,.(m) 
by which the reset service can deliver a message m from the user at node v to the user 
at node wu. There is also a FREEM,,, output action that is used by the reset service to 
indicate that it is ready to accept another message from node u to node v. Thus the 
external interface between a reset service and its users essentially mimics the interface 
offered by a UDL (see Chapter 5) except that packets are replaced by messages. This 


is an important property that will be exploited later. 


However, the interface between a user and a reset service is richer than the interface 
between a user and a set of UDLs. This is because the reset service at node wu offers 
two additional actions: an input action REQUEST, and an output action SIGNAL,. 
Intuitively, the REQUEST, action is used by the user at node wu to request a network 


reset, and the SIGNAL, action is used to inform the user at node wu that a reset has 
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Figure 7.2: Interface specification for reset service 
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been completed. We will refer to any event of the form SIGNAL, as a signal event and 


any event of the form REQUEST, as a request event. 


7.3.2 Difficulties in Specifying the Reset Problem 


The ideal behaviors of a non-stabilizing reset protocol can be specified in terms of 
three properties: timeliness, consistency, and causality. Intuitively, a behavior is 
timely if, in the absence of reset requests, free events are delivered in constant time 
and sent messages are delivered in constant time. A behavior is consistent if there is a 
symmetrical mating relation between signal intervals at neighbors. Finally, a behavior 
is causal if reset signals are only caused by reset requests and reset requests result in 


reset signals. 


As in a UDL, we cannot guarantee that any message sent by a user from say u to 
v will be received at v unless (see Figure 7.2) this message is sent after u performs a 
FREEM,, action. If this is true (and no other message is sent from u to v between 
the free action and the send action), then we will say that the send action is safe. We 
will only require delivery of messages corresponding to safe message send events. The 


specification will allow other messages to be dropped. 


Any specification of the behaviors of a stabilizing reset protocol has to take into 
account three anomalies that do not occur in the specification of a non-stabilizing reset 


protocol: 


e The first message sent on any link may not be safe in that it may not be preceded 


by a free event. 


e Some of the initial messages that are delivered may be abnormal in that they 
do not correspond to any messages sent. It appears that no stabilizing reset 
protocol can guarantee that there is some suffix of every behavior that contains 


no abnormal message delivery.' 


e Some of the initial signal actions may not be “caused” by any reset request. 


TA stabilizing reset protocol can begin in a state in which all links can have arbitrary messages 
stored. There can be executions that contain no state in which all links are empty of messages. Any 


suffix of such an execution will have abnormal message deliveries. 
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We handle these difficulties as follows. We add the following condition to the 
timeliness property: within linear time of the start of any behavior of the reset protocol, 
all received messages are normal — i.e., correspond to some message sent. We weaken 
the mating relation so that it is possible to receive abnormal messages in a signal 
interval — however, normal messages received in a signal interval must have been sent 
in a mated interval. Finally, we relax the consistency property and ask only that 
causality holds in linear time after the start of a behavior. Precise definitions are 


given in the next subsection. 


7.3.3 Formal Specification of Reset Problem 


Recall the following notation. When we say that an event a; occurs within time ¢ 
in @ we mean that a,;.time < @.start+ t. When we say that a time ¢ is a constant, 
we mean that ¢t = ct; + c't, where c,c' are some scalar constants and ¢, and t; are 
the default node and link delays respectively. In this and following chapters, we will 
also use the following notation borrowed from complexity theory. When we say that 
t = O(n), we mean that t < cn, where c is some constant time and n is the number 
of nodes in the graph G. This reflects a linear dependence on the size of the input, 
if we consider the input to be the network graph. If we say that an event a; occurs 
within O(n) time in a behavior 8, we mean that a,.time — B.start = O(n). If we say 
that an event a; occurs within O(n) time after an earlier event a; in behavior 3, we 
mean that a;.time — a;.time = O(n); in this case, we will also say that event a; occurs 
within O(n) time before event a; in behavior @. Sometimes we will say linear time to 


mean an interval of time that is O(n) in duration. 


As in a UDL, we cannot guarantee that any message sent by a user from say u to 
v will be received at v unless (see Figure 7.2) this message is sent after u performs a 
FREEM,, action. If this is true (and no other message is sent from u to v between 
the free action and the send action), then we will say that the send action is safe. We 
formalize this restriction by defining what it means for a message send event to be 


safe. 


Definition 7.3.1 Consider any behavior 2 of a reset protocol that has the interface 
shown in Figure 7.2. We say that an action a; = SENDM,,,(m) is safe in B if: 


e There is some FREEM,,, action in PB before a; and 
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e No other action of the form SENDM,,»(*) occurs between a; and the FREEM,» 


action. 


Clearly we would eventually like every message that is received at u from say v to 
correspond to some message sent from v to u. We will require that there is at most 
one message in transit from v to u. This will make it easy to make a correspondence 


between received and sent packets. Thus: 


Definition 7.3.2 We say that event ag = RECEIVEM,,,(m) in @ is a normal receive 


at u from v iff: 


e There is some a; = SENDM,,.(m) in @ that occurs before a, in 8 and 


e There is no RECEIVEMy,.u(*) event between a; and a, in f. 


We will refer to the earliest SENDMy.(m) in 8 that occurs before a, (and satisfies the 


above properties) as the send corresponding to dx. 


Notice that if the sender ignores the free notification and sends a message m several 
times before it is received, we make a correspondence between the receive and the 


earliest send event. 


Specifying Timeliness 


Next, we formalize the timeliness property. We we say that a behavior is timely if it 
satisfies four conditions for every pair of neighbors u and v. First all receive events 
that occur after linear time in @ are normal. In other words, after at most O(n) time, 
each packet received corresponds to some packet sent. Also any normal receive event 
occurs within O(n) time after the corresponding send event. This is shown in Figure 


7.3. 


The second condition is that either the user at u periodically receives a free action 
indicating that the link to v is free or else a signal event occurs periodically at either 
u or v. Third, any message sent by v to wu is either delivered in constant time or else 
a signal event occurs (at either u or v) after the message is sent. The essence of the 


second and third conditions is that, in the absence of signal events, free events are 
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Figure 7.3: Normal message receipt after linear time: There is a send action at v corresponding to any 
message received at u after O(n) time. Also any normal receive event occurs within O(n) time after it is 


sent. 


delivered periodically and sent messages are delivered in constant time. Intuitively, 
signal events are caused by reset requests; if reset requests are made continuously, then 


the reset protocol cannot guarantee periodic free events or the delivery of messages. 


The fourth timeliness condition is important (for example in Chapter 8) in appli- 
cations. It says that signals cannot keep occurring at wu without also occurring at a 
neighbor v. More precisely, if a signal event occurs at u then a signal event must occur 
at v in linear time before or after the signal event at u. However, because the reset 
protocol can start in an arbitrary state, we relax this requirement and only ask that 


this property hold after a linear amount of time. 


Definition 7.3.3 We say that a behavior B is timely if for every pair of neighbors 


U,V: 


1. Normal Receipt of Messages: There is some constant c such that every every 
receive event that occurs at time greater than §.start + c-n is normal. Also if 
a; is any normal receive event and a; is the send corresponding to a;, then a; 


occurs within O(n) time after a;. 


2. Periodic Free Events: Consider any t-suffiz y of behavior B. Then in y either 
a FREEM,, occurs within constant time, or a signal event occurs at u within 


O(n) time, or a signal event occurs at v within O(n) time. 


3. Timely Message Delivery: Suppose a; is a safe SENDMu,»(m) event in B. 


Then after a; either a RECEIVEM,,(m) occurs within constant time, or a signal 
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event occurs at u within O(n) time, or a signal event occurs at v within O(n) 


time. 


4. Signals at a Node induce Signals at Neighbors: There is some constant c 
such that for every SIGNAL, event a; that occurs at time greater than @.start+ce-n 


there is a SIGNAL, event that occurs in linear time before or after a;. 


Specifying Consistency 


Before we formalize the consistency property, we formalize the notion of a signal in- 


terval at a node in a behavior. 


Definition 7.3.4 Consider a behavior G3. A signal interval at node u in GB is a con- 


tiguous subsequence of 3 that: 


e Begins with either the start of 8B or a SIGNAL, event and 


e Ends with etther the first SIGNAL, event that occurs after the start of the interval 
or (if there is no such SIGNAL, event) the interval ends with the end of 8. In 


the latter case we call the signal interval a final interval. 


Thus any behavior induces signal intervals at each node. We now specify the 
consistency condition by requiring a mating relation between signal intervals. However, 
the mating relation is weaker than the relation for Data Links because it does not 
require the prefix property. The third property is a little tricky. Consider Figure 
7.4. Suppose the send at v is safe and is followed by a free action at v that is in the 
same signal interval at v. Then we require that the sent message is delivered and the 
delivery event occurs between the send and the free events. In essence, this states that 
all messages (except possibly the last message) sent safely in a signal interval at v are 


delivered at wu. 


Definition 7.3.5 We say that a behavior @ satisfies the consistency property if for 
every pair of neighbors u,v there is a symmetric relation (called a mating relation) 


between signal intervals at u and v such that: 
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Safe Send Action 
Both the send and the free 
occur in the same signal 


: interval at v. 
Free Action that 


indicates that messages 
can be sent to u. 


Figure 7.4: Successful sending of messages: between the sending of a message and the free action that 


indicates that the next message can be sent, either the message Is delivered or a signal occurs at the sender. 


1. At most one mate: A signal interval at u can be mated to at most one signal 


interval at v. 


2. Signal Intervals that communicate are mates: Let a, be any normal receive 
event at u from v in B and let a», be the send event corresponding to a,. Then the 
signal interval at u containing a, is mated to the signal interval at v containing 


Am: 


§. Successful Sending of Messages: Consider any safe SENDM,,.(m) event a; 
and a later FREEM,, event that occur in a signal interval at v. Then between 
these two events in B there must be a normal RECEIVEMy.(m) event a, such 


that a; 1s the send corresponding to ag. 


4. Mating of Final Signal Intervals: A final interval at u can only mate to a 


final interval at v. 


5. Mating Relation Preserves Temporal Ordering: Suppose a signal interval 
S, at u is mated to a signal interval S, at v and a signal interval Si at u is mated 


to a signal interval S! atv. Then if Si, occurs later than S, then S\, occurs later 


than S,. 


Suppose S, and S, are signal intervals at u and v respectively that are mates. No- 
tice that as compared to a Data Link specification, we have weakened the requirements 
for a mating relation: we no longer require that the sequence of message received by v 


from wu is a prefix of the sequence sent from u to v. However, if all received messages 
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are normal and all sent messages are safe, then the third consistency condition does 
imply the prefix property. The third and fourth consistency conditions also imply that 
if S, is a final interval, and if all received messages are normal and all sent messages 


are safe, then the two sequences are identical. 


The reader may wonder whether it is sufficient to specify consistent behavior only 
in final intervals. In that case, the consistency condition is much simpler. However, if 
the stabilizing reset protocol is to be used as a tool to build other stabilizing protocols 
(which is what we do in the next two chapters), then we claim that the reset protocol 
must make some guarantees during non-final intervals. A typical user of the reset 
protocol will be constantly doing some form of checking (see Chapter 8 for example) 
and will make a reset request if the checking detects a violation. But if the reset 
protocols exhibits arbitrary behavior during non-final intervals, then the user may 
always detect violations. These in turn lead to continuous reset requests which prevent 
the forming of a final interval. In other words, final intervals are only guaranteed if 
the user stops making reset requests; but users may only stop making reset requests if 
the reset protocol guarantees some form of consistency during non-final intervals. As 
another example, some user protocols may periodically make reset requests to start a 
new phase of the protocols. Such protocols (Chapter 9 describes an example of such 


a protocol) never stop making reset requests! 


Specifying Causality 


A causal behavior satisfies two intuitive conditions: that signal events are only caused 
by reset requests and reset requests result in signal events. Ideally, any signal event 
must be preceded by a reset request that occurs a linear amount of time before the 
signal event. Notice that this guarantees that all signal events will disappear in linear 
time after the last reset request in a behavior. However, because the reset protocol can 
start in an arbitrary state, we relax this requirement and only ask that this property 
hold after a bounded amount of time. But a causal behavior should also ensure the flip 
side of the above condition: reset requests should result in signal events. A protocol 
that simply ignored reset requests would be useless. We specify the second condition 
by requiring that a reset request at a node u is followed in linear time by a signal event 


at node w. 
Definition 7.3.6 We say that a behavior 3 is causal if: 
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1. There is some constant c such that every signal event a, that occurs at time 
greater than B.start + c-n is preceded by a request event a; that occurs in linear 


time before ax. 


2. A SIGNAL, event occurs within linear time after any REQUEST,, event. 


We are now ready to describe the behaviors that should be produced by a reset 


protocol. 


Definition 7.3.7 We define the reset problem RP to be the behaviors 8 that are timely, 


consistent and causal. 


The following lemma is useful because it tells us that every suffix of a behavior in 


RP is also in RP. 
Lemma 7.3.8 If a behavior § is in RP, then any t-suffiz of behavior B is in RP. 


Proof: The proof consists mostly of looking at each property in the definition of RP 
and showing that if a suffix of @ does not have the property then neither does 8. The 
only tricky case is the property that all messages received after at most O(n) time 
from the start of a behavior are normal. However, this can be deduced from the fact 
that the send event corresponding to a normal receive event in @ occurs at most O(n) 


time before the receive event. Jf 


7.3.4 Alternative Specifications of Reset Problem 


Traditional definitions (i.e., [AAG87]) of a reset service define the correctness of a reset 
service in terms of the states of protocol P, the user of the reset service. Our definition 
is more modular because it focuses on the behavior of the reset subsystem without any 


reference to the states of the users of the reset service. 


Next, one might like a reset protocol to satisfy a much stronger consistency property 
than the one specified above. In the stronger condition, the mating relation between 
signal intervals would also be transitive.. For instance, suppose u, v and w are con- 


nected in a cycle such that u,v, and v,w, and w,uw are all neighbors. Also suppose 
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that a signal interval S,, at wu mates to a signal interval S, at v, S, mates to some S, at 
w, and S, mates to some interval S$! at wu. Then our consistency specification allows 
S,, to be different from S!,. However, transitivity requires that S, = S/. The stronger 
condition seems to capture the essence of network synchronization in that the signal 
events provide consistent time points across the entire network. However, we show 
in the appendix that our reset protocol (and the reset protocol of |AAG87]|) does not 
satisfy the stronger condition. The applications in this thesis do not need the stronger 


condition. 


Note that the weaker condition does imply that there is a transitive mating relation 


between final signal intervals at all nodes in the network. 


Also the specification of the behaviors of a stabilizing reset protocol is complicated 
by anomalies that occur at the beginning of the behavior such as abnormal messages 
and signals that are not “caused” by any requests. Given this difficulty, we might be 
tempted to specify the reset problem using suffixes of the behaviors of a non-stabilizing 
reset protocol. This results in a more elegant definition. However, if we choose that 
definition, then we do not know any reasonably simple proof technique to show that a 


protocol stabilizes to behaviors that are suffizes of a specified set of behaviors.” 


7.4 Overview of the solution 


In this section, we will give an overview of a stabilizing reset protocol. Our solution 
basically consists of stabilizing the reset protocol of [AAG87] using the method of 
Chapter 5. In the first subsection, we describe a simple reset protocol that is not 
stabilizing. In the next subsection, the problems of the simple reset protocol are used 
to motivate the main ideas behind our reset protocol. Next, we give an overview of 
the code. Finally, we end this section with an example execution of our reset protocol. 


The next section contains a formal description of our reset protocol. 


2Such proofs seem to involve showing that every initial state of an automaton A is a reachable state 
of another automaton B. But familiar inductive proof techniques (such as invariant arguments, progress 


metrics etc.) do not seem to suffice to show that one state is reachable from another state. 
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7.4.1 Problems with a Simple Reset Protocol 


Amazingly, the consistency condition for normal reset behaviors can be guaranteed 
by the following Simple Reset Protocol (SRP). This is a non-stabilizing protocol. So 
assume that this protocol begins in a state where all queues and links do not contain 


any messages. 


In the absence of reset requests, nodes are in the so called Ready state. In this 
state, any message sent from (say u to v) is queued by u in an outbound queue for 
the link. When the message arrives at v, it is stored in a buffer which we will call 
buffer, |u|. Eventually, the message stored in buffer,|u] is delivered to the user at v. 
Thus in the absence of reset requests, sent messages are delivered in FIFO order. In 
the following two paragraph description of the protocol, when we say “node u” we 


mean “the reset protocol at node u”. 


When node u receives a reset request (REQUEST,,), the reset protocol at u sends an 
ABORT packet to all its neighbors and goes into a special Abort mode where it waits 
until it gets an ABORT packet from all its neighbors. It does so by setting a boolean flag 
ack,,[v] for all neighbors v to indicate that it is expecting an ABORT packet from v. A 
node u that receives an ABORT packet in the Ready mode behaves in almost the same 
way as a node that receives a reset request — i.e., wu sends an ABORT packet to all its 
neighbors. However, in this case wu sets ack, |v] for all neighbors v except the neighbor 
v from which it received the ABORT packet. If u receives an ABORT packet from 
neighbor v and ack, |v] = true then u sets ack,|v] = false. As soon as ack,|v| = false 


for all neighbors v, u returns to the Ready mode and performs a SIGNAL, action. 


The consistency condition is guaranteed by two additional rules. When an ABORT 
packet from wu arrives at v, buffer,|u] is emptied. Second, no packet in buffer,|u] is 
delivered while v is in Abort mode and until v has performed any outstanding SIGNAL, 
action. Intuitively, sending an ABORT packet on a link and waiting until another 
ABORT packet returns on the link effectively “flushes” any old messages that were sent 
in previous signal intervals. Essentially, we send an ABORT packet between any two 
SIGNAL events at a node and we delay delivery of packets until an outstanding signal 
has been performed. This ensures that a signal interval at u “communicates” or mates 
with at most one signal interval inv. The Simple Reset Protocol (SRP) is similar to the 


Chandy-Lamport snapshot protocol [CL85] with ABORT packets replacing “markers”. 
The problem with SRP is that it can easily be placed in a state where it never 
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Figure 7.5: Two “bad” states of the Simple Reset protocol. The black dot before an edge indicates that 
a node is waiting for an ABORT on that edge. 


terminates — i.e., violates the causality property. Consider the topology shown in 
Figure 7.5. Suppose in the initial state there is an ABORT packet in the channel from 
u to v, and u is waiting for an ABORT from v (but not from w) before u can exit from 
the Abort mode. Nodes v and w are in the Ready mode, and all other links are empty. 
Next, assume that the ABORT packet arrives at v which causes v to enter ABORT state 
and send an ABORT packet to u and w. Assume that the ABORT packet sent to u 
arrives quickly and causes u to return to Ready mode. Notice that by the rules of 
the protocol, v does not expect an ABORT from u. The resulting state is shown in 
Figure 7.5. By symmetry, it is clear that the execution can remain in a cycle of states 


where an ABORT packet continuously cycles through the network. 


Now, since SRP is a non-stabilizing reset protocol, it may be possible to show that 
(after proper initialization) SRP will never enter such a “bad state”. If the network is 
static, then this is indeed true. However, if network can have links that can fail and 
recover (a so called “dynamic network”) then a series of link failures and recoveries 
can leave SRP in the “bad” state depicted in Figure 7.5. 


More importantly for our purposes, SRP does not seem like a suitable point of 
departure for constructing a stabilizing reset protocol. Specifically, there does not ap- 
pear any easy way to make SRP locally checkable. For instance, consider the topology 
of Figure 7.5. In a “good” state of SRP, if there is an ABORT packet in the cycle, 
there must be at least one other ABORT packet in the cycle, which is travelling in the 
“opposite” direction. This seems hard to check locally, even with the addition of a 
small amount of state. Instead, our point of departure is the AAG reset protocol of 
[AAG87]. This protocol works in dynamic networks and can be made locally checkable 
and correctable. We describe some more details of how the AAG protocol works and 


why it avoids the problems of the Simple Reset Protocol in the appendix. The ap- 
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pendix also contains a description of the changes we made to make the AAG protocol 
stabilizing. For the rest of this chapter, we will describe our reset protocol that is 
based on (but not identical to) the AAG protocol. 


7.4.2 The basic idea behind the Reset protocol 


Our reset protocol uses essentially the same idea as SRP for ensuring consistency — 
once again consistency is ensured by using ABORT packets to “flush” old packets (that 
were sent in a previous signal interval) from links. However, our protocol is much more 


conservative about allowing a node to to return to Ready mode. 


A global (but approximate) summary is as follows. The protocol responds to reset 
requests in three phases. In the first phase, ABORT packets are sent out from nodes 
that receive reset requests. Nodes that receive ABORT packets and are in the Ready 
mode, broadcast ABORT packets to their neighbors. Thus the first phase consists of a 
wave of abort packets that spreads outwards from a reset request much the same way 
as in SRP. However, in our protocol the abort waves create an abort forest as they 
spread outwards. Consider any node u. Node wu’s parent in the abort forest is the 
neighbor from whom uw last received an ABORT packet that caused u to leave Ready 
mode. If node wu left Ready mode because wu received a reset request, then u has no 
parent and uw is considered a root in the abort forest. Notice that the abort forest is a 


set of ad hoc abort trees that are created while reset requests are being processed. 


In the second phase, a node sends an ack to its parent when the subtree rooted at 
that node has stopped expanding. Thus, in the second phase a wave of acks flow up 
the abort trees to the roots. The first and second phases work in much the same way 
as the Dijkstra-Scholten termination detection algorithm [DS80]. As in the Dijkstra 
Scholten algorithm, some nodes may be in the first phase (forward propagation of 
ABORT packets) while other nodes may be in the second phase (sending acks up to 


parents). 


What distinguishes our protocol from the Dijkstra-Scholten protocol is that there 
is a third phase in our protocol. When a root of an abort tree receives acks from all 
its children, it starts a wave of READY packets that flows down the tree and allows all 
nodes in the tree to return to the Ready mode. Thus the crucial difference between 


SRP and the AAG protocol is as follows. In the former, nodes return to Ready mode 
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after communicating with their neighbors. However, in our protocol a node u returns 
to Ready mode only after the root of the abort tree (that u is part of) knows that the 
abort wave has stopped propagating, and has informed wu of this fact using a READY 
packet. Thus in the SRP protocol a node returns to Ready mode in constant time after 
it receives an ABORT packet. In our protocol a node may have to wait O(n) time to 
return to Ready mode (potentially three worst case delays across the network, one for 


each phase). 


The reader may wonder why three phases are needed. The appendix provides some 
intuition by describing why three phases were used in the original AAG protocol to 
avoid the problems of the Simple Reset Protocol. 


7.4.3 Overview of the Code 


The code of our reset protocol works as follows. 


Each node wu has three interesting variables. First each node has a mode, mode, 
which is one of Ready, Abort or Converge. Ready, is the “normal” mode of a node when 
it is not processing a reset request. If the mode of u is Abort or Converge, then this 


means that u is processing an abort request. 


Next, each node u has an ack bit ack,[v] for each neighbor. If this bit is set, it 
indicates that u is waiting for an ack from v. (Unlike the Simple Reset Protocol, 
our protocol uses explicit ACK packets.) Finally, u has a parent variable parent,, that 
points to the neighbor from which wu received the reset request that it is processing. If 
u received a reset request through a REQUEST, action (i.e., a reset request directly at 
u itself) then uw sets parent, = nil. If parent, = nil, we will say that u is a root of an 


abort tree. A list of variables used by the code is shown in Figure 7.6. 


In our code, the mode of a node is characterized by the state of the other variables 


at the node. Thus the mode of node u is really a derived variable: 


Abort, if dv such thatack,|v] = true 
mode(u) =< Converge, if parent, # nil and Vv (ack,|v] = false) 
Ready, if parent, = nil and Vv (ack,[v| = false) 


Notice that unlike the Simple Reset Protocol, we have a third mode called Converge. 
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State 


ack,[v]: Boolean Flag for each neighbor v of u 

parent,: Either one of the neighbors of u or nil 

dist,: integer in the range 0..n’, where n’ is an upper bound on the number of nodes in the graph. 
We also assume that for any (ABORT, d) packet, d is an integer in the range 0..n’. 

signalbit,: Boolean flag (“used to remember to do a signal event*) 

free, [v]: Boolean Flag for each neighbor v of u (*says whether link to v is ready to accept a packet*) 
freem,,[v]: Boolean Flag for each neighbor v of u (*says whether v is ready to accept a message*) 
queue,,[v]: queue of size 5 consisting of packets drawn from Pata (*outbound queue for link to v *) 


buffer,,[v]: queue of size 1 that can contain a D-message only (*inbound message queue for link from v *) 
The following piece of code is used as a macro to propagate ABORT packets: 


PROPAGATE, (w, dist) = 
parent, := w 
dist, := dist 
For all neighbors v of u do (*broadcast abort packets to neighbors*) 
ack,|[v] := true 
enqueue (ABORT, dist+ 1) on queue,,|v] 


Figure 7.6: Variables and a Useful Macro 
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If mode(u) = Converge, this means that u has received acks from all its children and 


is waiting for a READY packet from its parent. 


There are also three interesting packets used by the protocol: abort packets (that 
are encoded as by tuples of the form (ABORT,d) where d is a integer representing 
distance from the root), ack packets (that are encoded simply as ACK), and ready 
packets (that are encoded simply as READY). 


The protocol will deadlock if the protocol is placed in an initial state in which the 
parent pointers form a cycle. In order to be able to locally check that the subgraph 
induced by the parent pointers is acyclic, we maintain a distance variable at each node, 
such that a node’s distance is one greater than that of its parent. Specifically, distance 
is initialized to 0 upon a reset request, and its accumulated value is appended to any 
abort packets. Thus we encode an abort packet as a tuple (ABORT, d), where d is a 


distance. 


The code that implements most of the reset protocol is shown in Figure 7.7. 
For convenience, we have marked certain transitions in the figure with the labels 


VR, VA, DA, IA, FA, RA and RR. 


A VR (for Valid Request) transition is a reset request that causes a node to change 
its mode to Abort. A VA (for Valid Abort) transition is the receipt of an (ABORT, d) 
packet with valid distance field that causes a node to change its mode to Abort. A 
DA (for Distance Invalid Abort) transition is the receipt of an (ABORT, d) packet such 
that the distance field dis at the maximum value and such that its receipt causes a 


node to change its mode to Abort. 


An IA (for Invalid Abort) transition is the receipt of an (ABORT, *) packet that 
does not cause a node to change its mode to Abort. A FA (for Final Ack) transition 
is the receipt of an ACK packet that causes say node u to send an ACK packet to its 
parent. It is not hard to see that the ack that was received must have been the last 
ack that node u was waiting for. A RA (for Root Ack) transition is the receipt of an 
ACK packet at a root node that causes the root node to change its mode to Ready. A 
RR (for Regular Ready) transition is the receipt of an (READY) packet at a node that 


causes the node to change its mode to Ready. 
Refer to these labels in Figure 7.7 when following the description below. 


The code in Figure 7.7 uses a small macro called PROPAGATE, (v,d). This proce- 
dure is used to broadcast (ABORT, *) packets and is shown in Figure 7.6. The first 
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Actions to Execute Reset Protocol 


REQUEST,, (*receive a reset request from user at node u, input action*) 
If mode(u) = Ready then 
VR:  PROPAGATE,(nil, 0) (*broadcast abort packets to all neighbors and set parent, = nil*) 
RECEIVE, ,14( ABORT, dist) (*receive abort packet from neighbor v, input action*) 
If buffer,,[v] is not empty then 
Empty buffer.,[v] (*flush any old message in buffer*) 
Enqueue &-Ack in queue,,[v] (*send a message ack allowing v to send another message*) 
VA: If mode(u) = Ready and dist < n’ then 
PROPAGATE,,(v, dist) (*broadcast abort packets to all neighbors*) 


DA Elself mode(u) = Ready and dist=n’ then  (*distance at max value; become a root and ack*) 
PROPAGATE, (nil, 0) 


Enqueue ACK in queue,, |v] (*send back an ACK as well*) 
IA: Else enqueue ACK in queue,,[v] (*send back an Ack*) 
RECEIVE ,x4(ACK) (*receive ack packet from neighbor v, input action*) 


If ack,|[v] = true then 


ack, |v] := false 


FA: If mode(u) = Converge then (*no acks outstanding and not a root?*) 
enqueue ACK in queue, (parent, | (*ack parent*) 
RA: — Else if mode(u) = Ready then (*no acks outstanding and a root?*) 
signalbit, := true (*remember to do a signal later*) 

For all neighbors z of u do 
Enqueue READY in queue, |z] (*broadcast READY*) 
RECEIVE, ,1(READY) (*receive Ready packet from neighbor v, input action*) 
RR: If parent, =v and mode(u) = Converge then (*Ready expected and from parent?*) 
parent, := nil (*return to Ready mode*) 
signalbit,, := true (*remember to do a signal later*) 

For all neighbors z of u do do 
Enqueue Ready in queue,,|z] (*broadcast READY*) 
SIGNAL, (*deliver reset signal to user at u*) 


Preconditions: signalbit, = true 


Effects: signalbit,, = false 


Figure 7.7: Actions at node wu to execute reset protocol functions with respect to any 
neighbor v. 176 


parameter to the procedure specifies that v is the new parent of u and the second 
parameter specifies that dis the distance from wu to root of u’s abort tree. The abort 
packets sent out as a result of this procedure (see Figure 7.6) will carry a distance 
value of d+ 1. We add the distance variable to abort packets in order to make the 
protocol locally checkable. 


When a reset request is made at some node u while in Ready mode (VR), u changes 
its mode to Abort, broadcasts an ABORT packet to all its neighbors, and sets its ack 
bits to true for all neighbors v. Node u then waits until all the neighboring nodes send 
back an ack packet. If node wu receives an abort packet while in Ready mode (VA), it 
marks the neighbor from which the packet arrived as its parent, broadcasts ABORT, 
and waits for ACK packets to be received from all its neighbors. We also add a check 
to see whether the distance in the ABORT packet is less than n’, which is the maximum 
value of the distance variable at a node. In linear time after all local predicates hold, 
this condition will always hold. However, this check helps to ensure that the local 
predicates remain closed during initial periods. If the distance check fails, (DA) node 
u becomes a root and sends out abort packets just as if it received a reset request; 
however, node wu also sends back an ACK to the ABORT packet it received. Finally, if 
the ABORT packet is received by a node not in Ready mode (JA), an ACK is sent back 


immediately. 


When node w receives an ACK from v it sets the ack bit for v to false. The action 
of node u when it receives the last anticipated ack depends on the value of u's parent. 
If w’s parent is not nil (FA), u’s mode is changed to Converge, and an ACK is sent to 
the parent. Notice that since mode, is a derived variable, mode, = Converge when 
ack,|v| = false for all neighbors v of u. If u is a root (RA), u changes its mode to 
Ready (by setting parent, = nil), and broadcasts a ready packet to all neighbors. If 
node u gets a READY packet from its parent while in Converge mode (RR), then u 
changes its mode to Ready (by setting parent, = nil), and broadcasts a ready packet 
to all neighbors. 


Finally, whenever node u changes its mode from either Abort or Converge to Ready 
it sets a flag (which we call signalbit,) to remind itself to later output a signal event. 
In other words, a SIGNAL, event is enabled whenever signalbit, is set; naturally, when 
this event occurs, the flag is cleared. For convenience, we introduce the following 


definition. 
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Definition 7.4.1 We say that node u has a status of on whenever signalbit, = false 


and mode, = Ready, and off otherwise. 


The code that implements the sending and receiving of /-messages is shown in 


Figure 7.8. 


If the status of u is on, u relays -messages between the user and the network. More 
specifically, when a SENDM,,,(m) event occurs and node wu has status on, wu queues m on 
an outbound queue (called queue,,|v] as in previous chapters) that contains packets to 
be sent to v. This queue can also contain an abort, ack or ready packet. Eventually, 
packets in this outbound queue are delivered to neighbor v. When v receives a »- 
message m from v, v queues m in an inbound message buffer called buffer,|u|. Later, 
if v’s status is on, it will do a RECEIVEM,,,(m) event and remove m from the buffer 
and deliver it to the user. See Figure 7.9 for a sketch of the inbound and outbound 


queues for a link. 


If u’s status is not on, u discards 4: messages input by the user through SENDM,,, 
actions. Also, u, will not do a RECEIVEM,, action unless its status is on. Recall 
that all %-messages from a neighbor v are queued on buffer,,|v]. In order to separate 
packets that are sent during separate signal intervals at v, we use the same rule as 
the Simple Reset Protocol. When a abort packet arrives from neighbor v, buffer,,|v| 
is emptied. Once again, this simple rule is really the key to ensuring the conditions 


required to satisfy the consistency property. 


Note that buffer,|v| can store at most one message. Clearly, if user messages are 
not to be dropped, we have to rely on the FREEM action as a form of “flow control”. 
Our scheme is as follows. We require that there is at most one L-message in transit 
from u to v. Thus u keeps a variable freem,,[v]. Whenever v delivers (or destroys) a 
&-message from u, v sends a &-Ack back to wu. When wu receives a -Ack from v, u sets 
freem,,|v] to true. This enables the FREEM,|v] action, which tells the user at u that 
it can safely send a message to v. Thus all we are doing is using a }-Ack message to 
provide feedback to the sender that the buffer at the other end is empty. 


7.4.4 Example 


Figure 7.10 describes a sample execution of the reset protocol for a simple topology 


consisting of four node u,v, w and «z connected as shown in the figure. The figure 
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Actions to send and receive 4-messages 


SENDM,,.(m), m € & (*input action by which user sends a message*) 
If mode(u) = Ready and signalbit,, = false and freem,,|v] = true then 
(* accept message only if mode is ready, no outstanding signals, and message flow control says OK*) 


enqueue ™ in queue,,|v] (*enqueue m in outbound queue to v *) 
freem,,|v] = false (*inhibit delivery of free event to user*) 
SENDz,0(p) (*output action to send a packet on UDL link to v*) 
Preconditions: 
pis the head of queue,,[v] (*p is head of outbound queue for link*) 


free, [v] = true 
Effects: 
Delete p from queue, |v] 


free, |v] := false 


FREEy,» (*input action from link to v to say it is free*) 
free, |v] := true 
RECEIVEy,x4(m), m € X (*input action to receive a user message from v*) 
Enqueue m in buffer.,[v] (*store m in inbound buffer for link*) 
RECEIVEy,1(2-Ack), (*input action to receive a X- message ack from v*) 
freem,,|v] = true (*record that v is ready to accept a new message*) 
RECEIVEM,,,(m), m € & (*output action to deliver a user message from v to the user*) 
Preconditions: 
m is the head of buffer,,[v] 
mode(u) = Ready and signalbit,, = false (*mode is ready and no outstanding signals*) 
Effects: 
Remove m from buffer,,[v] 
Enqueue »-Ack in queue,,[v] (*send a message ack back to v*) 
FREEM,,» (*output action to tell user that it can safely send another message to v*) 
Preconditions: 


freem,,|v] = true and signalbit,, = false and mode(u) = Ready 
Effects: 


None 


Figure 7.8: Actions at a node u to send and receive user messages to and from any neighbor v of u. 
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Message received from u at v 


Message sent from u to v 


queue ,[v], outbound packet queue 
can contain protocol packets and a 
user message 


buffer [u], inbound message buffer 
Vv 


Figure 7.9: Sketch of the inbound and outbound queues for a link. 


describes seven snapshots (F1-F7) taken during this sample execution. 


The execution begins (F1) with nodes u and w receiving a snapshot request. Node 
u and w go into Abort mode and send abort packets to their neighbors. In F2, the 
abort packet from u has arrived at v. Next, u and w receive each other’s abort packet 
and (F3) send back acks to each other. We assume that the abort from w to x travels 
slowly which allows the abort packet sent from v to arrive at z earlier (F3). Thus in 
F3, the abort tree rooted at u (which is shown using dotted lines) consists of u, v and 


az. 


Next, x sends an abort to both v and w which are acked immediately. Then z 
receives the “slow” abort from w and sends back an ack. Once x gets an ack back 
from both v and w, x sends back an ack to its parent v (F4). When v receives this 
ack, it sends back an ack to its parent wu (F5). Finally, in F6, u sends a ready packet 
down to v and then (F6), v sends a ready packet to z. Note that the ready packet 


destroys the abort tree as it travels down the tree. 


7.5 Reset Automaton 


In describing the reset code, we use the following notation. As in any node automaton 
there is a outbound queue of packets (queue,|v]) at any node w for every neighbor v 


of u. As usual, queue,|v| consists of packet waiting to be sent on Cy». 


Each node automaton WN, is specified given in Figures 7.6, 7.7, and 7.8. The code 
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Figure 7.10: A sample execution for the reset protocol described in terms of seven snapshots. 
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uses the variables and macro specified in Figure 7.6. The piece of code that deals with 
implementing the main part of the reset protocol is described in Figure 7.7. The piece 


of code that deals with sending and receiving -messages is described in Figure 7.8. 


7.6 Reset Protocol is Locally Checkable and Cor- 


rectable 


Let G be the topology graph that models the topology of the network on which the 


reset protocol works. 


Lemma 7.6.1 N = {N,,u € G} is a set of node automata for G with Pista = 4U 
{(ABoRT, *), ACK, READY}. 


Proof: Simple checking of the definitions given earlier. J 


7.6.1 Overview of Predicates 


Next, we prove that the protocol is locally checkable, by describing a set of predicates 
in Figure 7.14 and showing that these predicates are closed. The description of the 
predicates uses the shorthand notation shown in Figure 7.13. Please refer to both 


these figures during the following discussion. 


Recall that Q,,, is the queue corresponding to the single packet that can be stored in 
channel C,,,,. For convenience, we define zqueue,|v] (i.e., the extended queue between 
u and v) as the queue formed by the concatenation of Q,, and queue,|v]. Thus in 
Figure 7.9, the extended queue between u and v is the queue formed by concatenating 
the outbound link queue and the link itself. Note that we do not include the inbound 


message buffer! If Q., 4 nil, we assume that Q,,, is the head of zqueue,|v]. 


We informally describe the predicates. The first two predicates A and B deal with 
the ack flag ack,|v| at a node wu. Intuitively, this bit is set if w expects an ack from v. 
A states that if u is expecting an ack from v then one of three possibilities must be 
true: either there is an abort packet in transit from wu to v (Case 1 in Figure 7.11), OR 


v has received the abort packet and has chosen wu as its parent (Case 2 in Figure 7.11), 
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ABORT ACK | 


C) Vv mode(v) = abort 'e Vv OC Vv 


and v’s parent is u 


Case | Case 2 Case 3 


Figure 7.11: The Three Cases for the first and second predicates. The black dot before an edge indicates 
that a node is waiting for an ACK on that edge. 


OR there is an ack packet in transit from v to u (Case 3 in Figure 7.11). Predicate 
B says that at most one of these three possibilities can be true at the same time. In 


some sense, A and B govern the first two phases of the reset protocol 


Consider Figure 7.10. In that example execution, the first case is illustrated in F2, 


the second case in F3 and F4, and the third in F5. 
The next two predicates C and D govern the second and third phases of the reset 


protocol. They deal with the parent, variable at a node u. Intuitively, parent, = v 
if v is the parent of u in the abort tree. C states that if v is the parent of u and the 
the mode of u is Converge then one of three possibilities must be true: either there is 
an ack packet in transit from u to v (Case 1 in Figure 7.12), OR v has received the 
ack packet and has cleared its ack bit for wu but has not changed its mode to Ready 
(Case 2 in Figure 7.12), OR there is a ready packet in transit from v to u (Case 3 in 
Figure 7.12). Predicate D says that if v is u’s parent, then at most one of these three 


possibilities can be true at the same time. 


As an example, consider nodes x and v in Figure 7.10, where v is the parent of x. 
In that example execution, the first case is illustrated in F4, the second in F5 and F6, 


and the third in F7. 


The next predicate € is crucial to the proof of termination of the protocol. It states 
that the distance of a child in the abort tree is one more than the distance of its parent. 
The only exception to this is if the parent has “abdicated” by sending a ready packet 
that is currently in transit to the child. Essentially, this predicate shows that abort 


trees are acyclic and have a maximum height of n, the number of nodes; the proof of 
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mode(v) not equal to ready 


Vv . Vv 
ACK | READY 
©) u C) if @ u 
Case 1 Case 3 
Case 2 


Figure 7.12: The Three Cases for the third and fourth predicates. Node v is the parent of Node u in all 
three cases. 


termination will essentially consist of induction on the height of the abort trees. The 
predicate F is a supporting predicate required to prove that € is closed. It states that 
if u sends an abort to v then the distance in the abort packet is one more than u’s 


current distance. 


Predicate G is is again a supporting predicate required to prove that some of the 
other predicates are closed. Suppose there is a packet p, that is either a ready packet 
or a user message or a »-Ack, in transit from u to v. Then p must have been sent 
when u was in the Ready mode. Thus, it must be that either wu is still in the Ready 
mode or that u has gone into Abort mode since the time p was sent. But in the latter 


case there must be an abort packet behind p in transit from u to v. 


Predicate 1 governs the flow control scheme that ensures at most one message in 
transit from u to v. If freem,|v] is true then the FREEM,,, action is enabled and so 
there must be no message in transit from u to v and also no message acks (X-Ack’s) 
in the reverse direction. On the other hand, if freem,,|v] is false then either a message 
is in transit from u to v, or a message ack is in transit in the reverse direction, but not 


both. 


Predicate Q is the reason why we can get away with an outbound queue size for 
links (e.g., the size of queue,|v]) of 5. It states that the outbound queue (even after 
concatenation with the channel queue) can contain at most one packet of each type: 
abort, ack, and ready. Since #1 tells us we can have at most one U-message and at 


most one &-Ack, it means that a queue size of 5 is sufficient. 
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The following shorthand is used to specify local predicates: 


aqueue,,[v] (the extended queue between wu and v) is the queue 


formed by the concatenation of Q,,, and queue, |v]. 


Al(u, v) = (ABORT, *) in equeue,, [v] 

A2(u, v) = mode(v) = Abort and parent, = u 
A3(u, v) = ACK in zqueue, [u] 

C1(u, v) = AcK in zqueue,,[v] 

C2(u, v) = ack, [u] = false and mode(v) # Ready 
C3(u, v) = READY in aqueue, [u] 


Figure 7.13: Shorthand Used to Define Local Predicates 


We now show that the conjunction of these predicates is a closed predicate. The 


proofs, which we relegate to the appendix, consist basically of rigorous (and somewhat 


tedious) case-analysis. 


7.6.2 Proving that the Local Predicates of the Reset Protocol 


are Closed 


First, the the local predicates of Figure 7.14 are specific to a directed link (u,v). Recall 


that for local checkability we need exactly one local predicate for each edge. Hence: 
Definition 7.6.2 We let G,, be the intersection of the local predicates in Figure 7.14 


for any edge (u,v). For any edge (u,v), we let Ly, be the intersection of Gu», and 


Definition 7.6.3 Let L,, be the local predicate defined in Definition 7.6.2. Let L be 
the link predicate set containing Ly, for every (u,v) in G and let L = Conj(L). 
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A: ack,|v] = true iff 
one of Al(u,v), A2(u, v), or A3(u, v) holds. 


B: At most one of Al(u,v), A2(u, v), or A3(u, v) holds. 


C: parent, = v implies 
mode(u) = Converge iff 
one of Cl(u, v), C2(u, v), or C3(u, v) holds. 


D: parent, = v implies 
at most one of C1(u,v), C2(u,v), or C3(u, v) holds. 


E: parent, = v implies 
one of the following holds: 
dist, = dist, + 1 and mode(v) # Ready OR 
C3(u, v). 


F: If zqueue,,[v] contains a (ABORT, d) packet then d = dist, + 1. 


G: Ifpin zqueue,[v] and p= READY or p is a D-message or p = -Ack then 
Either mode(u) = Ready 
Or there is an ABORT in zqueue, |v] after p. 


H: Let M,,, denotes the concatenation of zqueue,[v], and buffer, [u]. Then: 
If freem,,|v] = true then 
There is no b-message in M,,, and no b-Ack in zqueue, [ul 
Else one of the following holds: 
There is exactly one 4-message in M,, OR 


There is exactly one -Ack in zgueue, [u] 


Q: squeue,,|[v| contains at most one (ABORT, *), ACK, or READY packet. 


Figure 7.14: Reset Protocol: Local Predicates for edge (u,v). Refer to the code given in Figure 7.13 for 
an explanation of the shorthand used. 
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The proof, which is in the appendix, consists of showing that each L,,, is a closed 


local predicate. 


Lemma 7.6.4 For a leader edge (u,v) and transition (s,a,s') of R, if s satisfies Ly., 
then s’ satisfies Ly». 


Proof: By lemmas, D.1.4, D.1.5, D.1.6, D.1.7, D.1.8, and D.1.3 in Section D.1 of the 
appendix. The predicates are closed because of the code in [AAG87] and the heuristic 


of removing unexpected packet transitions. 


We quickly sketch what is involved in such a proof. Consider predicate A as 
sketched in Figure 7.11. We essentially consider all states that satisfy this local predi- 
cate and show that no transition can cause this predicate not to hold in the next state. 
For example, consider Figure 7.11. If u is not expecting an ack from v in a state, then 
the only transitions that can cause this to happen is if uw gets a reset request or an 
ABORT packet while in Ready mode. But this causes u to send an an ABORT packet 


to v which leaves us in Case 1 of the predicate. 


Rather than consider all possible transitions, we save some effort by first identifying 
the transitions that can affect key variables. Then we focus our attention on such 


transitions. This method is described in the appendix. Jf 


The following theorem is immediate from the last lemma. 


Theorem 7.6.5 The reset automaton R can be locally checked for L using L. 


7.6.3 Reset Protocol is Locally Correctable 


Consider the function f defined in Figure 7.15. Let < be the trivial partial order such 
for all u,v,w,a in G, {u,v} < {w,c}. (ie., no pair of neighbors is less than any other 
pair of neighbors.) We claim that f is a local reset function for R with respect to L 
and partial order <. 


Lemma 7.6.6 The function f defined in Figure 7.15 is a local reset function for net- 
work automaton R with respect to link predicate set CL = {Ly} and partial order 
<. 
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Local Reset Function f applied to node u with respect to node v 


(*First simulate the receipt of an ACK message from v*) 
If ack,[v] = true then 
ack,|[v] := false 
If mode(u) = Converge then 
enqueue ACK in queue,,[parent, | 
Else if mode(u) = Ready then 
signalbit,, := true 
For all neighbors z of u do Enqueue READY in queue, |z] 


(*Next simulate the receipt of an READY message from v*) 
If parent, = v and mode(u) = Converge then 
parent, := nil 
signalbit,, := true 
For all neighbors x of u do do 
Enqueue READY in queue, |z] 


(*Finally correct parent, if it has not been done by code above*) 
(*and also clean up the message buffer and packet queue*) 

If parent, = v then parent, = nil 

Empty queue, [v] and buffer, [v] 


freem,,|v] = true 


Figure 7.15: Reset Protocol. Local Reset Function 
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Proof: Consider any state s of R and any leader edge (u,v) of G: We show each of 


the properties required by a local correction function. 


e Correction: In the state f(s|u,v), it is easy to check that parent, # v, ack,[v] = 
false and queue,|v| and buffer, |v| are empty. Also freem,|v] = true. Similarly, 
in the state (f(s|v,u), it is easy to check that parent, # u, ack,|u] = false and 
queue,(u| and buffer,|u] are empty and freem,|u] = true. Thus it follows that 


(f(slu,v), nil, nal, f(s|v,u)) € Lup. 


e Stability: We need to show the following fact for any neighbor w of v: 
If (s|u, \(u,v),8|(v,u),s|v) © Lu then (s|u, 8|(u,v),3|(v,u), f(slv,w)) © Luv. 


But if we look at the code for f(s|v,w) we see that f(s|v,w) is the same state 
that would occur at v in an execution in which the reset protocol starts in state 


s and consisting of the following sequence of actions: 


— A RECEIVE,,,(ACK) action (see Figure 7.7) followed by 
— A RECEIVE,,,(READY) action (see Figure 7.7) followed by 


— A hypothetical internal action that results in the execution of the last three 


lines of Figure 7.15 applied to node v with respect to node w. 


Now we know that the state at v resulting after the RECEIVE,,.( ACK) and 
RECEIVEy,2( READY) events must satisfy the stability condition because we have 
already proved this in Lemma 7.6.5. 


Now consider the hypothetical internal action applied to v. Note that this in- 
ternal action can only change parent, from w to nil. Also a careful look will 
show that this internal action will never change the value of mode(v) since it is 
only applied after the simulated processing of a READY packet from w. Now the 
predicates described in L,,,, only depend on the value of mode(v), ack,|u] and the 
predicate parent, = u. But none of these values are affected by the hypothetical 


internal action. Thus L,,, is unaffected by the hypothetical internal action. 


Thus the result of executing all three actions must satisfy the stability condition. 
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Lemma 7.6.7 Let f be the reset function described in Figure 7.15. Then R 1s locally 


correctable to L using link predicate set L, reset function f, and partial order <. 
Proof: Follows from Lemma 7.6.6 and Lemma 7.6.5. J 


Theorem 7.6.8 R* stabilizes to the behaviors of R(tp)|L in time t,, where t, and ty 


are constants. 


Proof: Follows directly from the Local Correction theorem, Theorem 5.4.3 and 
Lemma 7.6.7. Notice that height(<) = 1 since < is the trivial partial order. J 


7.7 The behaviors of a reset protocol after it sta- 


bilizes 


We have already shown that R* stabilizes to the behavior of R(t,)|L in constant time. 
Recall that R(t,)|L is identical to the automaton R except that the node and link 


delays are increased by a constant factor, and all link predicates hold. 


We now show that R(t,)|L solves the reset problem RP. However, because the set 
of behaviors specified by RP remains unchanged after scaling by constant factors, it 
suffices to show that R|L solves the reset problem RP. Thus in the remainder of this 
section and in the appendix, we show that every behavior 8 of RIL is in RP. 


We relegate the proof of this theorem to the appendix. However, in this section, we 
intuitively explain why any behavior @ of RL is timely, consistent, and causal. The 
intuition provided in this section should help the reader understand the proof in the 


appendix. 


In the proofs when we talk of a state s or an execution a, we mean a state or 
execution of R|L. Recall that we defined a derived variable called status(u) which is 
on if mode(u) = Ready and signalbit(u) = false and off otherwise. From the code, it is 


easy to see that u will only send and receive messages when status(u) = on. 


The first important lemma is what we call the Termination Lemma. This states 
the following. Consider any node wu. Assume that mode(u) # Ready in some state s; of 
any execution a. The lemma states that mode(w) will change to Ready in O(n) time 


after s;. 
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7.7.1 Why the Termination Lemma Works 


Let c be the worst-case time for a packet queued at node wu to reach neighbor v. It’s 
not hard to see that c is a constant because the outbound queue size at each node 
is at most 5 and because of the properties of a UDL. In the following, we will say 
that a node u is a root in some state s of the reset protocol if mode(u) = Abort and 


parent, = nilin state s. 


By assumption, mode(u) # Ready in state s;. Since mode(u) # Ready, u must 
either be a root or have a parent (say v) in state s;. By using invariants A and C 
repeatedly, we can obtain a chain of nodes starting with wu such that each node is the 
parent of the previous node. Also the chain must either end with a READY packet 
or must end with a root node r such that mode(r) = Abort in s;. This is shown in 
Figure 7.16. We also know from invariant € that the distance of each node in the chain 
is one more than that of its parent. Thus no node can occur more than once in this 


chain and hence this chain can consist of at most n nodes. 


(The following is a more detailed argument that explains why each chain must end 
in a READY packet or a root r. If mode(u) = Abort, then either u is a root or u has 
a parent, say v. In the latter case by A, ack,|u] = true and so mode(v) = Abort. If 
mode(u) = Converge then u has a parent, say v. By C, either there is an ACK in 
transit from u to v (in which case by A, mode(v) = Abort), or mode(v) # Ready, or 
there is a READY packet in transit from v to u. Thus either wu is a root, or u has a 
parent v such that mode(v) # Ready or there is a READY packet in transit from v to 
u. We now repeat this argument until we either arrive at a root node or a READY 


packet.) 


Consider the case where the chain ends with a READY packet. In this case, it is 
not hard to see that within O(n) time after s;,a READY packet will reach u which will 


cause u to go into Ready mode. 


So consider the latter case where the chain ends with a root r. Suppose we can 
show that in O(n) time after s;, there is an action a; in which r changes its mode to 
Ready and sends a READY packet to all its children. But if we can show this we are 
done in O(n) time after a; using the arguments in the previous paragraph. So all we 


have to do is to show that in O(n) time after s;, r will change its mode to Ready 


If the mode of r is Abort, then by definition r must have some neighbor v that it 


expects an ack from (i.e., ack,[v] = true). Now by invariant A, this means that (see 
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READY y 


OR 


u u 


Figure 7.16: The two cases that can occur if u’s mode is not Ready. Each node in a chain is the parent 


of the node below it. 


OR 


ABORT j ACK f 


Abort Chain Ack chain 


Figure 7.17: The two cases that can occur if node u is expecting an ack on the link to v. 
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Figure 7.11) either there is an abort in transit from r to v, or an ack in transit from v 
to r, or v is also in Abort mode and r is the parent of v. But if v is in in Abort mode, 
we can continue this argument inductively to produce a chain of nodes starting with r 
such that each node is the parent of the next node. We also know from invariant € that 
this chain can consist of at most n nodes. At the end of this chain (see Figure 7.17, 
there must be either an abort or an ack. We call a chain that ends with an abort 


packet an abort chain and we call a chain that ends with an ack packet an ack chain 


Recall that c is the constant that reflects the worst-case time for a packet queued 
at node to reach a neighbor. Now, observe that in c units of time any abort chain must 
either increase in size by 1 or be converted into an ack chain. But since the size of an 
abort chain cannot increase beyond n (the number of nodes), within O(n) time any 
abort chain must have converted into an ack chain. Similarly in c units of time, any 


ack chain must decrease in size by 1. Thus in O(n) time any ack chain will disappear. 


The upshot is that within O(n) time, ack,[v] will become false. Since this happens 
for any child v of r, within O(n) time after s;, r will change its mode from Abort to 


Ready and we are done. 


A formal proof can be made based on these arguments. The appendix contains a 


formal statement of the lemma (Lemma D.2.1) but omits a detailed formal proof. 


7.7.2 Why behaviors of the reset automaton are timely, con- 


sistent, and causal 


Recall that we defined a derived variable called status(u) which is on if mode(w) = 
Ready and signalbit(u) = false and off otherwise. From the code, it is easy to see that 


u will only send and receive messages when status(u) = on. 


The first important tool is what we call the Signal Lemma. Consider any execution 
a of R|L and any node wu and any state s; in a. The signal lemma basically says that if 
status(u) = offin state s;, then a SIGNAL, event occurs within O(n) time after s; in a. 
This follows because if signalbit, = true in s;, then a SIGNAL, event must occur within 
constant time of s; by the timing conditions. On the other hand, if mode(u) 4 Ready 
in s;, then the Termination Lemma tells us that in O(n) time after s;, we reach a state 


s; in which mode(u) = Ready. If s; is the first such state after s;, then by the code, we 
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know that u cannot change its mode to Ready without also setting szgnalbit, = true. 


Then, as before, a SIGNAL, event must occur within constant time of s; in a. 


Now we consider an execution a of R|L and sketch why the behavior corresponding 


to @ satisfies the timeliness, consistency, and causality properties. 


Proving the Timeliness Property 


Consider first the timeliness property: 


1. Normal Receipt of Messages: We need to show that there is some constant c 
such that every every receive event that occurs at time greater than @.start+c-n 
in any execution a is normal. Also if a; is any normal receive event and a; is the 


send corresponding to a;, then a; occurs within O(n) time after a;. 


This follows because that any message m in transit from say v to u (i.e, stored 
either in the queue at v, the link from v to wu, or the buffer at wu) cannot remain 
in transit for more than O(n) time. If a message m is stored either in the queue 
at v or in the link from v to u, then (by the properties of the link and the fact 
that the queue holds at most 5 packets), the message m will be stored in the 
buffer at wu in constant time. Next, we argue that in O(n) time after a state s; 
in which a a message m is in the buffer at u, m will either (see Figure 7.18) be 
delivered or be “flushed” by an ABORT packet sent from v. 


If status(v) remains on for a constant amount of time after s;, then the message 
m will be delivered. Thus the only other possibility is that message m remains 
in the buffer because status(v) is off in constant time after s;. But in that case 
by the Signal Lemma, there will be a SIGNAL, event in O(n) time after s;. With 
a little work (see appendix) we can show that such SIGNAL, events cannot keep 
occurring at node wu for a linear amount of time without causing v to send an 
ABORT packet to u; this will flush the buffer at wu. 


2. Periodic Free Events: Consider any t-suffix y of execution a. Then in ¥y either 
a FREEM,., occurs within constant time or a signal event occurs at u within O(n) 


time or a signal event occurs at v within O(n) time. 


This follows because of the following observation. Let c be a sufficiently large 


constant time. Suppose either status(u) or status(v) is off in c time after the 
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Figure 7.18: Two Cases for how a message is removed from a link 


start of y. Then we are done by the Signal Lemma. But if this is not true, 
and c is sufficiently large, then any message m in transit from u to v will be 
delivered in constant time; also any }-Ack message in transit from v to u will be 
delivered in constant time; this will result in freem,,|v] = true in constant time. 
In constant time after freem,,|v] is true (and assuming that c is large enough so 


that status(u) remains on in this interval), a FREEM,,, event will occur. 


. Timely Message Delivery: Suppose a; is a safe SENDM,,(m) event in +. 
Then either a RECEIVEM,,,(m) occurs within constant time after a; or a signal 
event occurs at wu within O(n) time or or a signal event occurs at v within O(n) 


time. 


This follows because of a similar observation to the one used to show periodic 
free events. Let c be a sufficiently large constant time. Suppose either status(u) 
or status(v) is off in c time after a;. Then we are done by the Signal Lemma. 
If not, by arguments similar to the ones above, we show that message m will be 
accepted and stored at u and then sent to v where it will be delivered in constant 
time. We assume that c is large enough so that status(u) and status(v) remain 


on in this interval. 


. Signals at a Node induce Signals at Neighbors: There is some constant c 
such that for every SIGNAL, event a; that occurs at time greater than @.start+e-n 


there is a SIGNAL, event that occurs in linear time before or after a;. 
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Figure 7.19: A signal event at u that occurs sufficiently “late” must be preceded in linear time by the 


sending of an ABORT packet to v. This in turn causes a signal event to occur within linear time at v. 


First, within linear time of the start of the execution a corresponding to §, there 
must be some state s, in which mode = Ready (by the termination lemma). In 
constant time after s,, there must be some state s; in which signalbit, = false. 
This follows because signalbit, cannot remain true for constant time without 
a SIGNAL, action occurring, which sets signalbit, = false. Thus any SIGNAL, 
action a; that occurs after state s; must have been “caused” by wu receiving a 
reset request or an ABORT packet while in Ready mode. Let this action be a,;); 
by the termination lemma, a,j occurs in linear time before a;. But as part of 
action a,, the code will also send an ABORT packet to neighbor v as sketched in 
Figure 7.19. Thus in constant time after a;, this ABORT packet will arrive at v 
and result in a state in which status(v) = off. Thus, by the Signal Lemma, in 
O(n) time after aj, a SIGNAL, event occurs. Since a, occurs within linear time 


before a;, the SIGNAL, event occurs within linear time before or after a;. 


Proving the Consistency Property 


Consider now the consistency property. As in the Simple Reset Protocol, the consis- 
tency conditions follow from the sending of ABORT packets between signal intervals. 
We define a signal interval S,, at u and a signal interval S, at v to be mates iff a nor- 
mal message sent in S,, 1s received in S, or vice versa. We can show that the mating 


relation is well-defined and symmetrical because of the following two properties. 


The first property which we call send consistency states that messages sent in a 


signal interval at v can be received in at most one signal interval at u; conversely 
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Figure 7.20: What happens between the sending of a message and a later Signal event. 


messages received in a signal interval at wu could have been sent in at most one signal 
interval at v. This will help establish that each signal interval at wu can have at most 


one mated signal interval at v. 


Send Consistency: Let a; and a, be any two normal receive events at wu from v 
in 8. Let a and a,, be the send events corresponding to a; and a, respectively. Then 
there is a SIGNAL, event between a; and a,, iff there is a SIGNAL, event between a; 


and ay. 


Consider Figure 7.20. This figure shows that if there is a SIGNAL, event after a, 
then there must be an ABORT packet sent in between from v to u. On receipt of this 
packet, all packets in the buffer at u will be flushed and status(u) must become off. 
Next, wu will not deliver any messages until it has performed a SIGNAL, and status(u) is 
on again. But any messages sent by v after the SIGNAL, event will arrive (by the FIFO 
property of the queue and link) after the ABORT packet arrives at u. This guarantees 
that such messages will not be delivered until u performs its SIGNAL, event. Similar 


(but more complicated) arguments can be used to show the other side of this claim. 


Next, we show a second property which states (in essence) that a signal interval at 
u cannot send messages to and receive messages from different signal intervals at v. 


This will help show that the mating relation is symmetric. 


Send-Receive Consistency Let a; be a normal receive event at u from v and 
let a, be a normal receive event at v from u. Let a; and a, be the send events 
corresponding to a; and a,, respectively. Then there is a SIGNAL, event between a; 


and a, iff there is a SIGNAL, event between a; and ag. 
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Consider Figure 7.20. This figure shows that if there is a SIGNAL, event after a, 
then there must be an ABORT packet sent from v to u before v performs its SIGNAL, 
event. When this packet arrives at v there are two possibilities. Suppose the mode of u 
is not equal to Ready at this point. If u has sent any message m to wu in the past, then 
u must have sent an ABORT packet to v after message m. On the other hand, if the 
mode of u is Ready when the ABORT packet from v arrives, then wu will immediately 
send an ABORT packet to u. In either case, the ABORT packet from wu to v will arrive 
at v before the ACK (see Figure 7.20 which shows the second case) from u to v. But 
when the ABORT packet arrives at v, it will flush out any messages in transit from u 


to v; it is only later that the SIGNAL, event can be performed. 


This effectively means that any messages sent by u after the ABORT is received at 
u can only be delivered at v after the SIGNAL, event. Similar arguments can be used 


to complete the proof of this claim. 
Now we show the third property of a consistent behavior listed in Definition 7.3.5. 


Successful Sending of Messages: Between any safe SENDM,,,(m) event and a 
later FREEM,,, event, there is either a RECEIVEM,,,(m) event or a SIGNAL, event. 


Call the first SENDM,.(m) event a;. If in state s;, status(v) = off, message m will 
be dropped; but in this case by the Signal Lemma a SIGNAL, will occur after s;. But 
in the period between s; and the SIGNAL, event, since status(v) = off, the code ensures 
that no FREEM,,, event can occur. So suppose m is placed in quewe,|u] in s;. Now 
we return to Figure 7.18. We know that either (case 1) m is delivered to u or m is 
destroyed by a later ABORT packet. In either case, we know from predicate H, that 
freem,|u] will be false until m is no longer in transit and so no FREEM,,, can occur in 
the interim. In Case 2, after the ABORT is sent by v, status(v) will remain off until a 


SIGNAL, occurs. Thus in this period as well no FREEM,,, event can occur. 


Next, we show the fourth property of a consistent behavior listed in Definition 


7.3.5. 


Mating of Final Signal Intervals: Let a; be a normal receive event at u from 
v in a and q be the send corresponding to a;. Then there is no SIGNAL, event after 


a; iff there there is no SIGNAL, event after a. 


This follows from Figure 7.20. If there is a SIGNAL, after a; there must be an 
ABORT packet that will be received at u after a;. This will result in status(u) becoming 
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off after a;. The Signal Lemma now tells us that there will be a SIGNAL, event after 


a;. The reverse argument is similar. 


Finally, the fifth property of a consistent behavior (i.e., the mating relation pre- 
serves temporal ordering) follows in essence from the fact that the UDLs are FIFO 
links and the fact that ABORT packets sent between signal intervals flush the links and 


buffers of previously sent messages. 


Proving the Causality Property 


To show that the behavior corresponding to execution a is causal, we prove the two 


properties of a causal behavior: 


1. There is some constant c such that every signal event a, in a that occurs at time 
greater than @.start+c-nis preceded by a request event a; that occurs in linear 


time before ay. 


It is sufficient to show that there is some constant c such that the following is 
true: if we consider any interval [s;,5,] in which no reset request occurs and 
such that s,.time — s;.time < cn, then a, cannot be a signal event. The required 
property follows from this because it implies that if a, is a signal event, then a 


reset request must have occurred in linear time before ag. 


If we choose c large enough, then we can break the interval |s;,5,] into four 


‘| and [s,r,$,] such that: 


subintervals [s;, si], [si, 8], [83, 5; 


e Every node u has had mode(u) = Ready in the interval [s;,5]. (More 
precisely, for every node u there is some state s; in the interval [s;, s;;| such 
that s;.mode(u) = Ready.) 


e Every node u has had mode(u) = Ready in the interval [s;, s;]. 
e Every node u has had mode(u) = Ready in the interval [s;, 557]. 
e For any u, any signal event enabled in s, is guaranteed to occur in the 


interval [s;/, 8,_1]. 


Intuitively, we choose the first three subintervals to be long enough, so that every 
node will have had a chance to go to Ready once in each subinterval. The termi- 


nation lemma tells us that this can be done using subintervals whose duration 
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is O(n). Finally, the fourth subinterval needs to be sufficiently long so that any 
signal events enabled at the start will occur before the end of the subinterval. 
By the timing conditions (see code), this can be done using a subinterval whose 


duration is a constant. 


First, note that a node u can become a root (i.e. mode, = Abort and parent, = nil 
only through two events: first, by a reset request at node u; and second by 
receiving an (ABORT,n’) packet with the distance variable at the maximum 
value. We call the second transition, a spurious reset request. We first claim 
that a spurious reset cannot occur in any state in which the following bounded 
distance predicate holds: for all nodes u, if parent, = nil then dist, = 0. This 
follows because if there is an (ABORT, n’) in transit from say v to u we can apply 
A and B repeatedly to obtain a chain of nodes ending with a root node r. If 
dist, = 0 then by € and F we can arrive at a contradiction in that any ABORT 


packet must carry a distance less than n — 1. 


Next, we claim that all spurious reset requests disappear after the first subin- 
terval. This is because the bounded distance predicate must hold after the first 
interval. If any node u becomes a root after the first interval, since it was Ready 
at some time during the first interval, it must have become a root through either 
a reset request or a spurious reset request; but either of these transitions will 


also set dist,, = 0. 


Next, we claim that for every node wu, at the end of the third subinterval that 


mode(u) = Ready. This follows because of the following intuitive argument. 


Recall that a root of an abort tree is a node u with parent, = nil and mode(u) = 
Abort. Second, we claim that any roots of abort trees that existed at the start of 
the second subinterval can no longer exist by the end of the second subinterval. 
This is because (by choice of subinterval) any such root node must have changed 
its mode to Ready by the end of the second subinterval. But since no reset 
requests occur in the entire interval, and no spurious requests occur after the 
first subinterval, no new roots can be created after the first subinterval. Thus 


there are no roots by the end of the second subinterval. 


Now consider any node u. But if there are no roots at the end of the second 
subinterval, u cannot enter Abort mode in the third subinterval. This is because 


in any state s in which s.mode(wu) = ABORT there must be a root node. (This in 
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turn follows by applying predicate B repeatedly.) But we know that u changed 
its mode to Ready somewehere in the third subinterval. Also, u cannot change 
its mode from Ready to Converge or Abort without first going into Abort mode. 
Thus, wu must be in Ready mode by the end of the third subinterval. 


Finally, if no real or spurious reset requests occur in the fourth subinterval and 
every node is ready by the end of the third subinterval, then no signal event can 
occur at the end of the fourth subinterval. This is because we choose the fourth 
subinterval such that any signal actions enabled at the start of this subinterval 


would occur before the end of this subinterval. 


2. A SIGNAL, event occurs within c-n time after any REQUEST, event in a. 


This follows easily because immediately after the REQUEST, event, status(u) = 


off. The claim now follows from the Signal Lemma. 


Thus, as we show more formally in Theorem D.2.28, every behavior of R|L is in 


RP. This can be used to show: 
Theorem 7.7.1 R* stabilizes to the behaviors in problem RP in constant time. 


Proof: Follows directly from Theorem 7.6.8 and Theorem D.2.28. Jj 


7.8 Local Correctability and Dynamic Network Pro- 


tocols 


A dynamic network is a network in which faults are limited to link failures: links 
can crash and recover in arbitrary fashion. A dynamic protocol is a protocol that 
works correctly in dynamic networks. If we assume that the topology (and the list 
of neighbors of each node) eventually stabilizes, then any stabilizing protocol P will 
eventually work correctly in a dynamic network. This is because any finite sequence 


of link failures can only leave P in some arbitrary state. 


A large number of protocols have been designed for dynamic networks. Dynamic 
protocols are useful because the most common faults in real networks are link and 


node crashes. Many of these existing protocols have not explicitly been designed to be 
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stabilizing. However, in this section, we conjecture that a number of dynamic protocols 


can be made locally correctable. 


We start with the reset protocol described in [AAG87] on which the reset protocol 
described in this chapter is based. This protocol was originally designed for dynamic 
networks. Thus besides the actions described in this chapter, the protocol in [AAG87] 
had actions for link failure and recovery. Thus for every node u and every neighbor 
v of u, the protocol had an input action LINK_UP,,, (corresponding to the link to v 
coming up at node u) and an input action LINK_DOWN,,, (corresponding to the link 


to v coming down at node u). 


Next, consider the reset function (Figure 7.15) used in this chapter for the reset 


protocol. The reset function applied to a node v with respect to a neighbor wu is exactly: 


e The code performed in [AAG87] for a LINK_DowN,,, event, immediately followed 
by 


e The code performed in [AAG87] for a LINK_UP,,, event. 


In other words, we can obtain a local reset function by simulating a link failure 


immediately followed by a link recovery. Is this a coincidence? 


We present a rough (but incomplete) argument as to why this might work. First, 
consider the stability condition for local correctability. Consider any neighbor w of w. 
Suppose that the (w,w) subsystem is in a good state — i.e., in a state that belongs to 
Luw. Now when the link to v comes up or goes down, the original protocol had to 
preserve the stability of L,,,. Thus the code for the LINK_UP,,, and LINK_DOWN,, 


events preserves the stability of L,,.,. 


Now consider the correction condition. Clearly it is possible in the dynamic pro- 
tocol to have the link at wu go down simultaneously at both ends and then come up 
simultaneously at both ends. When the link comes up it should come up with no 


messages in the links and with both u and v in states that satisfy L,,,. 


Both arguments given above are incomplete. For example, consider our “proof” 
of the stability condition. We claimed that if the (w,w) subsystem was in a good 
state, the LINK_DOWN,,, events would preserve the stability of L,,,,. To make a more 


careful argument we have to add the following local extensibility condition. For every 
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state a © Lu, there is some valid global state s of the network, in which the (u,w) 
subsystem is in state a and the link from node wu to node v is considered to be up at 
node uw. Since s is a valid state of the original protocol, and the LINK_DOWN,,, can 
occur in state s, the result of taking the LINK_DOWN,,,, action must result in a new 
valid state s’. But since s’ is a valid state of the original protocol it must satisfy all 


local predicates, including Ly». 


It is possible to formalize the intuitive arguments by adding similar local extensi- 
bility conditions, and showing that the protocol in [AAG87] satisfies these conditions. 


We will not do so here. 


7.9 Summary 


The three main ideas in this chapter are as follows: 


First, we have given a new definition of the correctness of a reset protocol in 
terms of its external behaviors. We have seen that the resulting definition is, in 
some sense, a generalization of the synchronization guarantees offered by a Data Link 
protocol. However, while a Data Link protocol synchronizes two nodes, a reset protocol 
synchronizes multiple nodes. Notice that our definition specifies the behavior of the 
reset protocol when reset requests are continuously being made and not just in the 
event that there is a last reset request. The definition we used in our original paper 
([APV91b]) only specified the behaviors in the event that there is a final reset request. 
However, in trying to apply the reset protocol (for instance, in Chapter 8) we soon 


found a need for the present specification. 


Second. we have applied the Local Correction theorem to stabilize a version of 
the reset protocol described in [AAG87]. We had to make some subtle changes to the 


original protocol to make it locally checkable and correctable. 


Third, we have conjectured that many locally checkable protocols that work in 
dynamic networks can be made locally correctable. To obtain a reset function, we 
concatenate the code that the original protocol used for a link down event with the 
code used for a link up event. As another example, Spinelli [Spi88a] describes a virtual 
circuit protocol that works in dynamic networks. This protocol appears to be locally 
checkable and it appears that we can use the link up and link down code to create a 


reset function. Interestingly, Spinelli [Spi88a] makes his protocol stabilizing by sending 
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a message periodically from the source of the virtual circuit to the destination of the 
virtual circuit and back. He also uses a timer whose value is proportional to the 
maximum end-to-end delay in the network. If, as we conjecture, local checking and 
correction is applicable to Spinelli’s protocol, the resulting protocol will be simpler 


and faster than the one presented in [Spi88a]. 
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Chapter 8 


Global Correction Theorem 


The previous three chapters have been concerned with local checking and local correc- 
tion. This chapter marks an important transition as we move to local checking and 
global correction. For the next two chapters, we will study the use of the stabilizing 
reset protocol of Chapter 7 for global correction. In this chapter, we will prove a 
Global Correction Theorem. This theorem states that any locally checkable protocol 
can be stabilized in time proportional to the number of network nodes using global 
correction. Thus global correction removes the need for the original protocol to be 
locally correctable but pays a price in terms of stabilization time. In the next chapter, 


we will apply global correction to a simple synchronizer protocol [Awe85]. 


The focus of this chapter is the Global Correction Theorem. The Global Correction 
theorem should be contrasted with the Local Correction Theorem (Theorem 5.4.3) of 
Chapter 5. The extra price paid for Global Correction is a stabilization time that is 
proportional to the number of network nodes instead of the height of some underlying 


partial order. 


A second important idea contained in this chapter is the concept of one-way check- 
ability. One-way checking is a special case of local checking that results in simpler 
and more efficient stabilizing protocols. We illustrate the two ideas in this chapter by 
using a simple spanning tree protocol. The spanning tree protocol is locally checkable 
and hence can be stabilized using the Global Correction Theorem. However, because 
the Spanning Tree protocol is also one-way checkable, we can create a simpler (and 


more efficient) stabilizing spanning tree protocol. 
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This chapter is organized as follows. In the first two sections, we state and prove 
the Global Correction Theorem. In Section 8.3 we describe a locally checkable protocol 
to compute a spanning tree. After all local predicates hold, this protocol computes a 
spanning tree in time proportional to the diameter of the network. Because it does not 
appear that the spanning tree protocol is locally correctable, the methods of Chapter 
5 do not seem applicable. However, the Global Correction Theorem applies to this 


spanning tree protocol. 


Next, in Section 8.4 we explain the concept of one-way checkability, and show 
that the spanning tree protocol is one-way checkable. We combine this observation 
with with the Global Correction Theorem to yield a simple stabilizing spanning tree 
protocol. Finally, in Section 8.6 we quickly sketch how Global Correction can also be 
applied to the design of a stabilizing protocol for topology maintenance in dynamic 


networks. 


8.1 Statement of Global Correction Theorem 


In this section and the next, we show that any locally checkable protocol N can be 
efficiently stabilized using a reset protocol. The basic idea in the proof of the Global 
Correction theorem is extremely similar to the transformation used in Chapter 5 (see 
Figure 5.6 and Figure 5.7) for the proof of the Local Correction theorem. As before, 
we augment the original automaton with actions to perform local snapshots on every 
(u,v) subsystem. However, there are two crucial differences in the proof of Global 


Correction: 


e First, we replace all links (UDL’s) in the original protocol by a stabilizing reset 
protocol for graph G as described in Chapter 7. In other words, each node u 
now communicates with its neighbors using the interfaces provided by the reset 
subsystem. Now, the reset protocol offers an interface that is identical to a UDL 
except that packets are replaced by messages. Thus we also have to replace the 
actions that send and receive packets in the node automata with actions to send 


and receive messages. 


e Second, when a violation is detected in the (u,v) subsystem, a global reset request 


is made using the REQUEST, action offered by the reset protocol interface. Then 
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when a SIGNAL, action occurs, node wu will essentially do a “local restart” by 
initializing its state. This will guarantee that no more violations of the (u,v) 


subsystem will be detected after both u and v have initialized their state. 


Suppose we are given a locally checkable automaton NV. Then in the transformation 
outlined above, we need to replace the events to send and receive packets by events to 
send and receive messages. Thus we cannot guarantee that the transformed automaton 
stabilizes to the behaviors of NV|L but to a renamed version of N|£ in which action 


names have been suitably renamed. This motivates: 


Definition 8.1.1 Consider a network automaton N. We define R(N), the message- 
renamed version of N, to be the automaton that is identical to N except that for all 


U,V: 


e Every SENDu»(*) (i.e., packet send) event in N is renamed as a SENDMy»(*) 


(i.e., message send) event. 


e Every RECEIVE,,»(*) (i.e., packet receive) event in N is renamed as a RECEIVEM,,»(*) 


(i.e., message receive) event. 


e Every FREE,» (i.e., packet free notification) event in N is renamed as a FREEMy,» 


(i.e., message free notification) event. 


The message-renamed version of a set of behaviors of a network automaton is defined 


similarly. 


Note that renaming does not really affect the services offered by an automaton 
because the users of the automaton can change their interface to accommodate the 


renaming. 


With this definition, we can state the main theorem of this chapter. It states 
that any locally checkable network automaton N can be transformed into another 
automaton N+ such that N+ stabilizes to a version of NV in which i) All local predicates 
hold ii) The node and link delays are increased by a constant factor iii) The packet 


events are renamed as message events. 
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Theorem 8.1.2 Global Correction: Consider any network automaton N = Net(G, N) 
that is locally checkable for some predicate L using link predicate set L. Then there 
exists some N* that is a message-renamed UIOA for graph G and a constant c such 


that N+ stabilizes to the behaviors of R(N(c)|L) in O(n) time. 


8.2 Proof of the Global Correction Theorem 


We present the main ideas in the proof of the Global Correction Theorem in the 
following subsections. We first sketch the construction of the augmented automaton 
N+. Then, we show that after linear time of the start of any execution of V~ all reset 
requests disappear. We use this to conclude that NM stabilizes to a message-renamed 
version of AN in linear time. Finally, we make some observations about modularity in 


the proof. 


8.2.1 Construction of Nt 


Let Ly,» be the local predicate for each leader edge (u,v). We construct NV as follows. 
As in Chapter 5, for every leader edge (u,v) we add actions to nodes u and v to perform 
the local snapshot protocol on the (u,v) subsystem. The local snapshot protocol is 
used to detect a violation of predicate L,,. When the leader u detects a violation, u 
makes a reset request. The code is derived from the code used for the proof of the Local 
Correction theorem in Chapter 5 (Figures 5.6, 5.7, and 5.5) by making the following 


changes: 


e All sending and receiving of packets in the modified node automata Nj} is re- 
placed by sending and receiving of messages so that Nj can be a user of the 
reset subsystem. Thus instead of sending request and response packets, we send 
request and response messages. Similarly we have to replace the free packet 


notification events with free message notification events. 


e There is no longer a need for a mode variable in N,* because each phase is 
implicitly a snapshot phase. Whenever the mode is checked in the original code, 


we only follow the code path corresponding to mode = snapshot. 


e We add a requestbit,, variable to N;* to remember to do a reset request. 
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When a response is received at the leader and a local predicate violation is 


detected, requestbit, is set to true (instead of changing mode,|v] to reset). 


We compose the modified node automata N}{ with the reset protocol R|L of 
Chapter 7. Thus in the final automaton, N; has input action SIGNAL, and 
output action REQUEST,. 


Nj; makes a reset request REQUEST, whenever requestbit,, is true; on taking this 


action requestbit, is set to false. 


On receiving a SIGNAL, event, Nj does a local restart. It first resets the basic 
state of the original automaton N,, to some initial value I, ( we will discuss how 
I, is determined below). Also for each neighbor v of wu, uw will locally reset all 
the phase and free variables. More precisely, the free,,[v|, freeg,,|v| and phase,,|v| 
variables are all set to false for all neighbors v. The counter variable count,|v| 
is initialized to 1 (if u is the leader) and to 0 (if u is not the leader). This will 
ensure that after a signal event at u and v, all (u,v) phases are clean and all 


message send events at wu are safe. 


The definition of local checkability in Chapter 5 ensures that there is always at 
least one global state s which satisfies all local predicates. Now if there are any 
messages in transit in global state s, we consider the state s’ that results when all 
these messages are received by their destinations. Because the local predicates 
are all closed predicates, s’ satisfies all local predicates as well. But in addition 
s’ has no messages in transit. Then we choose the initial state [,, for each node 


u to be equal to s’|u. 


The modified code is described in Figure 8.1, Figure 8.2, and Figure 8.3. The 
message formats and timing partitions are as in the proof of the Local Correction 
Theorem. As before, we hide all actions of V* that are not actions of NV. 


8.2.2 All reset triggers disappear in Linear Time 


In the proof we will make heavy use of the reset protocol specification given in Chapter 
7 and some aspects of the proof of the Local Correction Theorem in Chapter 5. Refer 


to Chapter 7 for definitions of normal messages, the send corresponding to a receive, 
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SENDM,,,(p) (*output action for p € Paata only*) 
Preconditions: 
freeq,,|v] := true and free,,[v] := true 
p is head of queue, |v] 
((U(u, v) = u) and (phase,[v] = false)) OR (([(u, v) = v) and turn,|v] = data)) 
Effect: 
freeq,,|v] := false and free, |v] := false; 
Remove p from head of queue,,|v] 
turn,|v] = response 


(* give response packets a turn*) 
phase,,|v] = true 


(* start a new checking/correction phase*) 


FREEM,,y (*input action*) 


Effect: freeq,,|v] := true and free,[v] := true; 


Figure 8.1: Code for the modified SEND,,.(p) actions at a modified node N;t in order to do Global 
Correction. 
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SENDMu,v(Preq) (*output action: u repeatedly sends a request till it gets a response™) 


Precondition: 

I(u,v) =u (*u is the leader of link subsystem*) 
(phase,,|v] = true) or (queue,[v] is empty) (*phase in progress or no data waiting*) 
freeq,,|v] = true (* no message in transit on link to v *) 
Preq count = count,[v); (* counter correct*) 

Effect: 
freeq,,|v] = false (* set to false until link says it is free*) 
phase,,[v] := true; (*remains true until matching response returns*) 
RECEIVEMy,u(Preq) (*input action, receive request at u from v*) 

Effect: 
If p.eg-count # count,[v] and I(u,v) = v then (*not a duplicate or invalid message™*) 
count,[v] := Dreqg count; (*remember count*) 


Figure 8.2: Code to send and receive request messages at node u in order to do Global Correction. 
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SENDMu,v(Presp) (*output action: u repeatedly sends a response to last request*) 


Preconditions: 
I(u,v) =v (*u is not the leader of link subsystem*) 
(turn,|[v] = response) or (queue,,|v] is empty) (*response’s turn or no data messages waiting”) 
freeq,,|v] = true (* no message in transit on link to v *) 
Presp-count = count,|v]; 
Presp:-node_state = s|u 
Effect: 
turn,|v] := data (*give data messages a turn*) 
freeq,,|v| := false (* set to false until link says its free*) 
RECEIVEMy,u(Presp) (*input action to receive response at u from v*) 
Effect: 


If (county[v] = Presp-count) and (phase,,|v] := true) and (I(u, v) = u) then 
If (s|u, nil, nil, p,.,,-node_state) ¢ Ly then requestbit,, := true 
phase,|v]| := false, (*end of phase*) 
count, [v] := (count,[v] + 1) mod 4; 


REQUEST,, (*input action to make a reset request*) 
Preconditions: 
requestbit,, := true; 
Effect: 


requestbit,, := false; 


SIGNALy (*input action to receive a Signal at u*) 
Effect: 
For all neighbors v (*make links to all neighbors clean*) 


phase,|v]| := false, 
If (u,v) = u then count,[v] := 1 else count,|v] := 0 
Empty queue,|[v] and set freeg,|v] and free,,|v] to false 


sju= I, (*initialize node state*) 


Figure 8.3: Code to send and receive response messages and to make reset requests and respond to 


signals at node w. 
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causality, timeliness, and the mating relation. Refer to Chapter 5 for definitions of a 


phase and a clean link. 


The proof consists of showing that all reset requests will eventually stop and that 
in linear time after this ZL holds. Define a (u,v) reset trigger to be the receipt of 
a snapshot response from v at u that causes u to set requestbit, = true. The main 
part of the proof consists of showing that all reset triggers disappear in linear time 
(which by causality implies that all signals disappear). We start with some preliminary 


definitions. 


Consider an execution a of Nt. Define a quiescent state in a to be a state s; in a 
such that: 


e All messages received after state s; in a are normal. 


e All signal events that occur after state s; are preceded by a request event that 


occurs in linear time before the signal event. 


e For any pair of neighbors u and v, a signal event a; at u that occurs after state s; 
is accompanied by a signal event at node v that occurs within linear time before 


or after a;. 
We have the following lemma: 


Lemma 8.2.1 Quiescent Lemma: A quiescent state occurs within linear time of 


any execution of N*. 


Proof: NV* is the composition of the augmented node automata with R|L. We know 
the behaviors of R|L are timely and causal. The lemma follows from the first timeliness 


property, the first causality property, and the fourth timeliness property. [ 


Suppose that u is the leader of some (u,v) subsystem. We show that in any 
execution a of V+, there exists some linear time ¢ such that all (u,v) reset triggers 
disappear in time ¢ after a quiescent state in a. Since a quiescent state occurs in linear 
time after the start of a, it follows that all reset triggers disappear in linear time after 
the start of a. 


The proof is in two parts. First, we show that one of two cases must occur in linear 
time after a quiescent state. Next, we show that no (u,v) reset triggers can occur after 


the occurrence of either of the two cases. 
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One of Two Cases must Occur in Linear Time after a Quiescent State 


To define the two cases, we need two simple preliminary definitions. Recall from 
Chapter 7 the definition of a signal interval and the definition of a send corresponding 
to a normal message. Intuitively, an initialized signal interval at a node is a signal 
interval that begins with a signal event at the node; we give it this name because each 
node initializes its local state immediately after a signal event. Next, a regular message 
receipt is a message that was received in an initialized signal interval and was sent in 


an initialized signal interval. Formally: 


Definition 8.2.2 An initialized signal interval at node u is a signal interval at node 


u that begins with a SIGNAL, event. 


Definition 8.2.3 A message receive event is regular if 


e The message receive event is normal 

e The message is received in an initialized signal interval at the receiving node. 

e The send corresponding to the receive event occurs in an initialized signal interval 
at the sender. 


Next, we state the main lemma of this subsection. 


Lemma 8.2.4 Quiescent Cases: Consider any execution a of Nt and any leader 
edge (u,v). Within linear time after any quiescent state s; of a there is a state s; such 


that one of the following two cases is true: 


e Case 1: Any message received by u from v (and vice versa) after s; is regular 


OR 


e Case 2: In state s;, Ly, holds and (u,v) is clean. Also, the interval |s;,s;] is 
contained in some signal interval at u as well as some signal interval at v, and 


these two signal intervals are mates. 
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Proof: (Sketch) Recall the definition of a (u,v) phase from Chapter 5. Roughly 
speaking, a (u,v) phase is an interval that begins with the leader u sending a request 
and ends with u receiving matching response. Recall also the definition of a clean phase 
from Chapter 5. The proof also makes considerable use of the timeliness, consistency, 
and causality properties (see Definitions 7.3.3, 7.3.5, and 7.3.6) of the reset automaton 
RL. 

We choose state s; as some state that occurs after state s; and satisfies either one 


of two properties: 


a) Property 1: Six (u,v) phases occur in the interval [s;,s;] AND there are no 


signal events at u or v in [s;,8;] AND L,,, holds in state s,. 
OR 


b) Property 2: There is some state s, in the interval [s;,s;] such that at least 
one signal event occurs at both u and v before s,. Also, for any message receive event 
that occurs after s;, the corresponding send event occurs after s, (i.e., all messages in 


transit in state s, are delivered before state s;). 


We now show that we can find such a state s; that occurs within linear time after 
5; 

From the second and third timeliness conditions, we see that message delivery 
between u and v behaves exactly like a UDL in the absence of signal events at u or v. 
In other words, the reset subsystem delivers messages and free notifications in constant 
time in the absence of signal events. Thus from the proof of the Phase Rate lemma in 
Chapter 5 (Lemma 5.6.5), we see that in the absence of signal events at either u or v, 
a (u,v) phase will complete in constant time. Similarly, in the absence of signal events 
at either u or v, six (u,v) phases will complete in constant time. By Lemma 5.6.12, 
the sixth (u,v) phase is a clean phase — i.e., a phase in which the matching response is 
sent by v after the receipt of the request. Thus the sixth phase will produce a correct 
snapshot of the (u,v) subsystem. Thus by the end of the sixth phase, either L,,,, holds 
or node wu will make a reset request. But if node u makes a a reset request, then by 


the second causality property, a signal event will occur at wu in linear time after s;. 


We conclude from the last paragraph that in linear time after s; either we reach a 
state s; satisfying Property 1 or a signal event occurs at either u or v. But if a signal 
event occurs at either u or v in linear time after s;, then we know (from our choice of 


s; and the fourth timeliness property), that a signal event occurs at both u and v by 


215 


some state s, that occurs within linear time after s;. But in the latter case, the first 
timeliness property tells us that there is some state s; that occurs within linear time 
of s, and such that all messages “in transit” in state s, have been delivered before 
state s;. Thus in linear time after s; we reach a state s; satisfying either Property 1 


or Property 2. 


We are now ready to prove the lemma. If s; satisfies Property 2 we show that Case 


1 must be true; and if s; satisfies Property 1 we show that Case 2 must be true. 


If s; satisfies Property 2 then we know that all messages received at u from v (and 
vice versa) after state s; was sent after some state s,. We also know that at least 
one signal event has occurred at both u and v before state s,. Thus all such message 


receive events are regular, and we have Case 1 in the lemma statement. 


If s; satisfies Property 1 then we know by Lemma 5.6.12 that (u,v) is clean in 
state s;. We also know that L,,, holds in s; and that no signals occur at either u 
or v in [s;,s;]. Thus there must be some signal interval (say S,,) at u that contains 
the interval [s;,s;]. Similarly, there must be some signal interval (say S,) at v that 
contains the interval [s;,s;]. Now the sixth (u,v) phase in [s;, s;] is a clean phase. But 
in a clean phase, v sends a response to u after v receives a request from u. Thus there 
is at least one message sent after s; by wu that was received before s; at v. Hence, from 
the consistency property, the intervals S, and S, must be mates. Thus we have Case 


2in the lemma statement. J 


All (u,v) triggers disappear after either Case 1 or Case 2 


Next we show that for any leader edge (u,v), all (u,v) reset triggers stop after one of 
these two cases listed above occur. Since we have just shown that that one of these 
two cases must occur in linear time, we conclude that all reset triggers disappear in 


linear time. 


We first show all (u,v) triggers disappear after Case 1 occurs: 


Lemma 8.2.5 Case 1 Trigger Termination: Consider any execution a of N* and 
any leader edge (u,v). Suppose that after any quiescent state s; of a there is a state s; 
such that any message received by u from v (and vice versa) after s; is regular. Then 


no (u,v) triggers occur after state s; in a. 
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Figure 8.4: Correct Snapshots after nodes u and v have each performed a signal event. 


Proof: (Sketch) Recall that a (u,v) trigger is the receipt of a response at wu which 
causes u to set requestbit, = true. Consider the receipt of any response at wu after s;. 
By construction, this response must be received in an initialized signal interval (say 
S, at wu) and must correspond to a response sent in an initialized signal interval (say 


S, at v). This is shown in Figure 8.4. 


We claim that the sequence of messages received by u in S, is a prefix of the 
sequence of messages sent by v in S,, and vice versa. We will refer to this as the prefiz 


property. The prefix property follows from the consistency property because: 


e S, and S, are mates, 
e All messages received after s; are normal and 


e All messages sent in an initialized signal interval are safe. 


We now claim that that the matching response cannot be a (u,v) trigger because of 
the following argument. First, at the start of S,, u sets its basic state to [, and at the 
start of S,, node v sets its basic state to [,. But (Lu, nil, nil, I,) € Duy». Next, at the 
start of S,, count,|v] is initialized to 1; also at the start of S,, count,[u] is initialized to 
0. Thus by the prefix property, the sequence of states and messages sent and received 


in S, and S, could have occurred in some execution y of the (u,v) subsystem, such 
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that (u,v) is clean and L,,, holds in the first state of y. Thus, as in Chapter 5, any 
matching responses will carry accurate snapshot information, and will not result in a 


(u,v) reset trigger. I 


Note that what makes this and the result in Chapter 5 work is that L,, 1s a closed 
local predicate — once L,,, holds, it continues to hold, regardless of the behavior of 


other subsystems. Next, we show all (u,v) triggers disappear after Case 2 occurs: 


Lemma 8.2.6 Case 2 Trigger Termination: Consider any execution a of N* and 
any leader edge (u,v). Suppose that after any quiescent state s; of a there is a state 
s; such that: 


e Lu» holds and (u,v) is clean in s;. 


e The interval |s;,s;] is contained in some signal interval at u as well as in some 


signal interval at v, and these two signal intervals are mates. 


Then no (u,v) triggers occur after state 5; ina. 


Proof: (Sketch) The argument is extremely similar to that for Case 1 except in the 
signal intervals at both wu and v that contain the interval [s;,s;]. This is shown in 
Figure 8.5. Let the signal interval at u that includes state s; be called S,. Let the 
signal interval at v that includes state s; at be called S,. We know that S, and S, are 
mates by assumption, and that in s;, Dy, holds and (u,v) is clean. This is shown in 


Figure 8.5. 


Thus the sequence of states and messages sent and received in S, and S, after state 
s; could have occurred in some execution y of the (u,v) subsystem, such that (u,v) 
is clean and L,,, holds in the first state of 7. But then, any matching responses will 
carry accurate snapshot information, and will not result in a (u,v) reset trigger. Thus 
in the first signal intervals for Case 2 we rely on an explicit state s; (in which the local 
predicate holds and the link is clean). By contrast, in Case 1 we relied on the first 


signal intervals being initialized signal intervals. 


The argument for signal intervals after S, and S, in Case 2 is identical to the 


argument for Case 1 because such signal intervals are initialized signal intervals. [J 
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Figure 8.5: Correct Snapshots after Ly.) holds and link (u,v) is clean. 


8.2.3 All Local Predicates Hold and All Signals Stop in Linear 


Time 


From the Quiescent Lemma, the Quiescent Cases Lemma, and the Case 1 and Case 
2 Trigger Termination Lemmas, it follows that all reset triggers disappear in linear 
time after the start of any execution of V+. Hence, by causality, all signal events also 
disappear in linear time. More precisely, any execution a of V* has some O(n)-suffix 


yy in which there are no signal events. 


But now it is easy to see that in constant time in 7, all local predicates hold. We 
know from the second and third timeliness conditions (and the fact that there are no 
signals in y) that six (u,v) phases will complete in constant time after the start of 
y'. But after these six phases are complete, L,,, must hold. If not, wu would detect 
a violation in the sixth phase (which is guaranteed to be a clean phase) and u would 


have made a reset request. But this will result in a signal event in -y, a contradiction. 


Lastly any execution of M+ which contains no signal events and in which Ly, 
holds for all links can be shown to be an execution of N(c)|L for some constant c. As 
in Chapter 5, the c represents a constant slowdown due to the overhead of periodic 


checking. 
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8.2.4 Proof of Global Correction Theorem is Modular 


We note that in the transformation used to prove the Global Correction Theorem we 
used the reset protocol R|L from Chapter 7. However, the proof only uses properties 
of the behaviors of R|L and thus R|L could have been replaced by any automaton 
that has the same interface as ®R|L and has the same behaviors. Further, note that 
the modified node automata Nj (that we create by the transformation) are UIOA and 
R\|L is a CIOA (since every reachable state is also a start state). Thus, the modularity 
theorem assures us that we can replace R|L by its stabilizing implementation (the 
automaton R* as described in Chapter 7) without affecting the Global Correction 


result. 


8.3 A Locally Checkable Spanning Tree protocol 


We begin by describing earlier stabilizing spanning tree protocols and their disadvan- 
tages. Then we describe a locally checkable spanning tree protocol. This protocol will 


be used in Section 8.5 to construct a stabilizing spanning tree protocol 


8.3.1 Previous Work on Spanning Tree Protocols 


The basic idea in virtually all spanning tree algorithms is that nodes report the smallest 
node ID seen so far (and the shortest distance to this smallest ID node) to their 
neighbors. Each node then picks as its parent the neighbor that knows of the smallest 
ID. If more than one neighbor reports the smallest ID, the node picks from among 
these the neighbor that reports the smallest distance. If two neighbors report the 
same distance and root ID, then an arbitrary tie-breaker is used to select the parent. 
A node sets its estimate of the root ID to equal its parent’s root ID, and updates its 
distance from the root to be one plus the parent’s distance. Because of the way the 
distance from the root is calculated, this approach is basically an adaptation of the 


Bellman-Ford Algorithm. 


However, in a dynamic network (and in a stabilizing setting), this approach en- 
counters an obstacle known as “ghost roots”. This phenomena occurs whenever the 


root crashes: its ID, which was the smallest in the system, is still the smallest from 
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the point of view of nodes that do not know of its crash. Even in a static network, 
the same effect can be caused by initial errors that introduce a root ID lower than 
the ID of any network node. This ID can potentially remain forever in the system! 
Even if the nodes maintain a counter to reflect their distance from the alleged root, 
this counter will simply grow unboundedly. Since the ghost root phenomenon is not a 


rare event, several ways to overcome this difficulty have been suggested. 


One solution, known as the “hop counter” approach, is to have some pre-determined 
bound on the diameter of the network at each node, and to discard the old root 
whenever the associated counter reaches some limit (see, e.g.,|/ AG90]). Unfortunately, 
this pre-determined bound must be quite high, and hence, the stabilization time of such 


counting up schemes is poor in practice [CRKG89a, CRKG89b, Gar89, RF89, Awe90]. 
Another widely used stabilizing Spanning tree protocol is the IEEE 802.1 bridge 


routing protocol which is based on the design in |[Per85]. This solution uses an approach 
that we call timer flushing. The basic idea [Per85] is that each node “times out” 
information received from its neighbors unless the information is refreshed periodically. 
Any node that thinks it is the root is responsible for periodically sending updates to 
all its neighbors in order to refresh their state information. Any other node X sends 
estimates to its neighbors only after receiving a message from the node X thinks is its 
parent. The upshot is that if there is an old root in the system, the information about 


this old root will eventually “time out”, after which the system will stabilize. 


However, timer flushing suffers from several drawbacks. First, the node clocks (by 
which packet lifetimes are enforced) can have different rates. Second, the message 
delivery time over links typically has large variance; since the topology of the network 
is not known in advance, the variance of the message delivery time across the whole 
network is even higher. A conservative timeout bound must take into account high 
message latencies and the worst-case topology, even though the latencies of an actual 
execution may be considerably (e.g., an order of magnitude) smaller. This leads to an 
order of magnitude slowdown of the stabilization process. Lastly, the parameters of 
such “global” timeout protocols often have complicated dependencies. Tuning these 


parameters is typically quite difficult. 


Th stabilizing spanning tree protocol we will describe in Section 8.5 is extremely 
similar to the two schemes we have described above. However, it uses a different 
mechanism for detecting and recovering from states with ghost roots that speeds up 


the stabilization time of the resulting protocol. 
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The detection mechanism is based on the following observation. Consider an exe- 
cution of the simple spanning tree protocol that starts with a state in which all nodes 
are correctly initialized and there are no messages in transit on the links. Now focus 
on some node. Throughout the execution the node maintains a current estimate of the 
root ID, and another estimate for its distance from this alleged root. It can be shown 
that in the course of legal executions, the node’s estimate of the root ID never goes 
up; also while such a root estimate is fixed, the distance estimate never goes up. This 
property can be cast in the form of a local predicate for each link. If the predicate 
holds, then the algorithm will produce a spanning tree. This immediately suggests the 
stabilizing algorithm: whenever the predicate is violated, the node that detects the 
violation makes a reset request. In the execution that follows the last reset signal, all 


the information will be correct. 


Rather than directly describe the final spanning tree protocol of Section 8.5, we 
will derive it in the following way. In the next subsection, we will describe a locally 
checkable (but non-stabilizing) spanning tree protocol on which the final spanning 
tree protocol is based. Of course, the Global Correction Theorem is applicable to 
this protocol. However, we will show in the following section that the spanning tree 
protocol is also one-way checkable. We then combine these observations to produce the 
final stabilizing spanning tree protocol in Section 8.5. The resulting protocol is simpler 


and more efficient than a spanning tree protocol based only on Global Correction. 


8.3.2 Code for Locally Checkable Spanning Tree Protocol 


We now describe the code for our locally checkable (but non-stabilizing!) spanning 
tree protocol. After all local predicates hold, this protocol computes a spanning tree 


in time proportional to the diameter of the network. 


Before we delve into the code of the protocol, we define what it means for a spanning 
tree protocol to be stabilizing. We assume that the network topology is described by 
some topology graph G and that there is an output action REPORT,(p) at every node 
u. This action is used to report the parent p of node wu in the spanning tree, where 
p is some neighbor of u in the network graph. We say that a spanning tree protocol 
is stabilizing if it stabilizes to the stable tree behaviors for graph G. The stable tree 


behaviors for graph G are the behavior £ in which for every node wu: 
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e For every t-suffix y of @ and for every node u, a REPORT, event occurs in constant 


time. (i.e., nodes report their parent values at constant intervals of time.) 


e For every node uw, if there is REPORT,(p;) event and a later REPORT,(p2) event 


in 2, then p; = po. (i.e., the parent values reported by nodes never change). 


e Consider any set of REPORT,.(*) actions in 8 containing exactly one REPORT, (pu) 
action for every node u. Then the graph induced by the values of the p,, variables 
is a spanning tree of G. (i.e., the parent values reported by nodes forms a 


spanning tree of G) 


Our locally checkable spanning tree protocol is described by the automaton T, 
shown in in Figure 8.6. The protocol is based on the simple idea alluded to earlier. 
Each node keeps track of the smallest node ID seen so far and the shortest distance to 
this smallest ID node. Each node uses this information to compute its parent in the 


tree. 


Thus each node u maintains a parent pointer parent, current estimate of the root’s 
identity r,, and a current distance estimate d,,. We denote by (r.,d,) the ordered pair 
at node u. We will use lexicographic ordering for 2-tuples and 3-tuples. For example, 
(rv, dy) < (ru, du) means that either r, < ry, or that r, = r, and d, < d,. Similarly, 
(ry, dy, parent,) < (ru, du, parent, ) means that that either (r,,d,) < (ru,du), or that 
(ry, dy) = (ru, d,) and parent, < parent,. Each node also maintains a copy (possibly 
outdated) of the root and distance estimates of each its neighbors. Thus the estimates 


of neighbor v are stored at u in the variables r,[v] and d,|[v]. 


For compactness, we let both arrays have an entry for the node itself, in which the 
default values are “hardwired”: thus r,/u] = u, and d,[/u] = —1. Now a node always 
chooses its own estimate based on the minimum of the estimates of all its neighbors 
and its own default estimate. Node u chooses its own estimate of the root from this 
minimum estimate; u also chooses its own estimate of the distance as one plus the 
distance from the minimum estimate. Thus if u itself is the smallest root, the distance 
u chooses for itself will be —1 + 1 = 0, which is as it should be. Thus d,|u] is set to 


—1 simply as a sentinel value to avoid an extra case. 


Every node wu periodically sends its own estimate of root and distance to all its 
neighbors using an “Announce” packet. To make the T,, a node automaton, we have 


followed the convention (see Chapter 5) of first enqueuing the “Announce” packet on an 
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outbound queue for the link called queue,,|v]; then the packet is sent out from the queue 
when the link is free. The Announce packet is encoded as a tuple (Announce,r, d). 
When a node wu receives an Announce packet from v, wu first checks whether the estimate 
in the packet is greater than the previous estimate stored from v and the distance in 
the estimate is not already at the maximum possible value. If this is not the case, u 
stores the latest estimate it has received from v; u also updates its own estimate as 


the minimum of all neighbor estimates. 


Let T,, be the automaton shown in in Figure 8.6. We assume that for any u,v, all 
actions of the form SEND,,,(*) are in a separate class, and any ENQUEUE,,,, action is 
in a separate class. Similarly for any w, all actions of the form REPORT,(*) are in a 


separate class. 


Let T be the composition of the automata node T, for every node in G with a 
UDL for every edge (u,v) in G. In the next subsection we show that T is locally 
checkable for a predicate L that we define. In the following subsection, we show that 
T|L stabilizes to the stable tree behaviors in time proportional to the diameter of 


graph G. This sets the stage for applying the global correction theorem to 7. 


8.3.3 Local Predicates for 7 


We state a set of local predicates for the spanning tree protocol. Let us denote by 
Ou,» the intersection of the local predicates shown in Figure 8.7 for any edge (u,v). 
Note that Ou, consists of two predicates, Ol(u,v) and O2(u,v). Ol(u,v) says that 
the sentinel values that node u uses (as a neighbor estimate from itself) are correct. 


It also says that node wu’s estimate is the minimum of all its neighbors. 


O2(u,v) shows that the sequence of estimates sent from u to a neighbor v are non- 
increasing. Suppose there is an (Announce,r,d) packet in transit from u to v, then 
the estimates that v already has stored from u can be no less than the estimates in 
the packet (i.e., (r,[u], d,[u]) > (r,d)). Also the estimates in the packet can be no less 
than the estimates at u (i.e., (r,d) > (ru, du)). If there is no (Announce, *, *) packet in 
transit from u to v, then the estimates that v already has stored from u can be no less 
than than the current estimates at u (i.e., (ry[u], d,[u]) > (ru, du)). Recall that a Unit 


Capacity Link only allows one outstanding packet on each link, just as in a UDL. 
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State 


neighborSet, set containing all neighbors of u and u itself 

parent,, parent pointer, in neighborSet,, 

di: distance estimate, integer in the range 0..n’, n’ is an upper bound on number of nodes. 
Py! root estimate, all root estimates are in the range of node identifiers 

Py |v]: estimate of r, for each v in neighborSet, except r,,[u] = u 

d,,[v]: estimate of d, for each v in neighborSet, except d,[u] = —1 

free,|v| boolean, true if link to v is free for each v in neighborSet, except wu itself. 

queue, |v] a queue containing at most one (Announce, x) packet to be sent to v. 

Actions 

FREEu,» (*input action to receive free notification from link to neighbor v*) 


Effects: free,|v] = true 


SENDy,»(p) 


(*output action to send estimate to neighbor v*) 


Preconditions: p is the head of queue,,[v] and free,|v] = true 


Effects: free,|v] = false; Remove p from queue,,|v] 


ENQUEUE,y 


(*output action to enqueue estimate to neighbor v*) 


Preconditions: queue, |v] is empty 
Effects: Add (Announce, ry, dy) to queue,|v] 


RECEIVEy,,( Announce, r, d) (*input action to receive estimate from neighbor v*) 
Effects: 
If (r, d) < (ru[v], du[v]) then (*received estimate is no greater than previous estimate”) 
If (d < n’) (*received estimate not at max value *) 
(ru[v], dulv]) := (7, d) (*update estimate from v*) 


(Tu, dy, parent,) := min{(ry|v], du[v] + 1,v),v € neighborSet,, } 


REPORT, (q) (*output action to report parent estimate”) 


Preconditions: g = parent, 


Figure 8.6: Spanning tree protocol. Code for a node w. 
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Definition 8.3.1 For any edge (u,v), we let Ly, be the intersection of Ou, and Oyu. 
Let £ be the link predicate set containing L.,, for every (u,v) inG and let L = Conj(L). 


It is easy to show that each L,,, is a closed local predicate. 


Lemma 8.3.2 For a leader edge (u,v) and transition (s,a,s') of R, if s satisfies Ly, 
then s’ satisfies Ly». 


Proof: First, Ol(u,v) is clearly closed because the code never changes the value of 
ry[w] and d,|w]. Also, if the code changes (ru, du, parent,,), it always sets this 3-tuple 
equal to the minimum of (r,[v], d.[v], parent, |v]) for all v in neighborSet, as required 


by O1(u,v). 


Next, O2(u,v) is closed because of a simple observation: the (r,d) estimates sent 
in (Announce,r,d) packets by node u are always non-increasing. This follows because 
node u changes its own estimate after receiving an Announce packet from some neigh- 
bor w. But (see code), wu will not accept the packet from w unless the new estimate 
from w is no greater than the previous value that u has stored from w and if the 
distance from w is not already at the maximum value n’. But if the new estimate from 
w is no greater than the previous value that u has stored from w, then the new esti- 
mate that u calculates will be no greater than before. But if O2(u,v) is true initially 
and all estimates sent by u are non-increasing, then it is easy to see that O2(u,v) 
remains true. Note that if u does not accept a packet from neighbor v (e.g., because 


the distance estimate in the packet is equal to n’) this does not affect O2(v, wu) 


Finally, since the above argument can be applied symmetrically for the link (v, w), 
it follows that L,, 1s closed. Jf 


Note that the check in the code that prevents v from accepting an estimate larger 
than a previously stored estimate is really an application of the heuristic of removing 
unexpected packet transitions (see Chapter 6). Clearly if u receives an estimate from 
w larger than the previous estimate stored from w, then O2(w,u) is not true in the 
state before the estimate was received. Thus this is an unexpected packet transition; 


by removing such a transition, we ensure that L,,,, is a closed predicate. 


The following theorem is immediate from the last lemma. 


Theorem 8.3.3 The network automaton T can be locally checked for L using L. 
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Ol(u,v): 
Py[u] = u 
d,|[u] = —1 
(Tu; dy, parent, ):= min{(ry[v], dy[v] + 1,0), v € neighborSet, } 


02(u,v): 
If there is an (Announce, r,d) packet in transit from u to v then 
(re[u], delul) > (7,4) > (rus du) 
Else (*there is no Announce packet in transit*) 


(ro[u], do[u]) > (7a. du) 


Figure 8.7: Spanning Tree Protocol: Local Predicates for edge (u,v). 


8.3.4 Fast Computation of Spanning Tree after all Link Pred- 
icates Hold 


We now show that 7|L stabilizes to the stable tree behaviors in time proportional 
to the diameter of G. Recall that T|L is the spanning tree automaton described in 
Figure 8.6 such that all local predicates hold in the initial state. 


The basic idea is to show first that if all local predicates hold then there can be 
no ghost roots; next, we show that if the spanning tree protocol begins in a state that 
does not contain a ghost root, then the protocol quickly converges to a stable spanning 


tree. 


We assume that each node has a unique ID that is modelled by the name of the 
node in the topology graph. Thus the unique ID of node u is wu. Our spanning tree 
protocol depends heavily on the fact that each node has a unique ID that cannot be 


corrupted. Thus we are assuming that the node ID is part of the code at a node. 


We first define what it means to have a ghost root in a state of J. Informally, 
consider the set of all the IDs present (in all the root estimates at nodes and in all 
the Announce packets in the links) in state s. If the minimal ID of all nodes in the 


network is not the the minimal ID in this set, then we have a ghost root. Formally: 
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Definition 8.3.4 Consider a state s of T. We say that s has a ghost root r ifr<u 


for all nodes u in G and either: 


e There is an (Announce,r,*) packet in transit from u to v in s for some u,v, OR 
e r,|v] =r ins for some u,v, OR 


er, =r ins for some u. 


The only way we can have ghost roots in any execution is because of bad data in 


links and nodes when the protocol starts up. 


Lemma 8.3.5 Consider any execution a of T. Suppose there is a state s; in a such 
that there is no ghost root in s;. Then then there is no ghost root in the state following 


8; in Q. 


Proof: From the code it is easy to see that the only way a new value is added to set 
of existing roots is if a node wu adds its own ID to the set (e.g., by changing its root 


estimate to the default). Thus ghost roots cannot be created after state s;. I 


Next, we show that any state in which all local predicates hold cannot contain a 


ghost root. 
Lemma 8.3.6 There is no ghost root in any state of T|L. 


Proof: Suppose we did have a ghost root in some state s of T|LZ. Then there must 
be a ghost root r with the lowest ID among ghost roots in s. If it is in an Announce 
packet in transit from say v to u, then we know from O2(v,u) that r, < r and hence 
ry =r. On the other hand, if there is some w such that r,|rz] = r or r, = 7, then we 
know from O1(w, *) that r, <r and hence r,, = r. Thus, we have some node, say w, 


such that r,, =r. 


By Ol(u,v) if parent,, 4 w we can continue inductively following parent pointers. 
Each time we move from say w to x = parent,,, we know from O1(w, *) that (ry, dy) = 
(rw[z],dw[e] +1). But since r, = r, (rwla],d,[z]) < (r,dy). But we know from 
O2(z,w) that that (r.,d,) < (rw[z], dw[xz]). Thus we conclude that (r.,d,) < (r, dy). 


228 


But the distance estimates are non-negative and the root estimates in this chain 
are all equal to r since r is the lowest ID ghost root. Hence the process of following 
parent pointers must terminate at some node say z such that parent, = z andr, =r. 
Thus by O1 applied to z, we know that r = z. But that contradicts the fact that r is 
a ghost root. Jf 


The last important fact to observe is that if there are no ghost roots and all local 
predicates hold, then the protocol converges in time proportional to the diameter of 
the network. As usual, we will say that ¢ = O(d) to mean t < cd where c is a constant. 


We denote the diameter of G by D. 


Lemma 8.3.7 Consider an execution a of T|L. Then there is some t-suffiz of ezxe- 


cution a (where t = O(D)) whose behavior is a stable tree behavior for graph G. 


Proof: We know from from Lemma 8.3.6 that there is no ghost root in the initial 
state of a and hence by Lemma 8.3.5 there is no ghost root in any state of a. From 
the fact that O2(u,v) is closed, we know that O2(u,v) holds for all links (u,v) in all 
states of a. Let g be the node with the smallest ID in graph G. We prove the lemma 


using induction on the distance d of a node u from gq. 


Inductive Hypothesis: There exists some constant c such that within c-d time 
after the start of a, there is some state in which (r,, dy, parent,,) = (q,d,v), where v is 
some neighbor of u at distance d—1 from qg. Also u will not change r,,d, or parent, 


from this state onwards in a. 


The inductive hypothesis implies the lemma because it implies that within time 
proportional to the diameter, every node has a parent pointer that points to a node that 
is one hop closer to the root q. It also implies that the parent pointers do not change 
after this point. Also, since for each u, all REPORT,(*) actions are in a separate class, 


such a REPORT,(*) action must occur in constant time in any suffix of an execution. 


Basis, d = 0: Node q is the only node at distance 0 from itself. In all states s; 
of a, (79,49, Pq) = (4,0,¢), or there would be a ghost root in state s; which we have 


already ruled out. 


Inductive Step: Suppose it is true for all nodes at a distance d from s and we 
wish to show that it is true for a node v at a distance d+ 1 from s. Thus there is some 


state s; such that all nodes u within distance d from q have (r., du, parent,,) = (q, d, *). 
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Also, s; occurs within c-d time after the start of a. We first claim that in all states 
of a, (ry, dy) > (q¢,d +1). Suppose this were not true in some state s, of a. Then by 
following parent pointers as we did in the proof of Lemma 8.3.6 and by using predicates 
Ol and O2 iteratively, we can show that we must end in a ghost root. Since we have 
ruled out ghost roots, (r,,d,) > (q,d+ 1) in all states of a. 


Next, v must have some neighbor at a distance d from gq. Let u be the neighbor 
with the lowest ID among the neighbors of v at distance d from g. Thus by the 
properties of a UDL, within constant time after state s;, an (Announce, q,d) packet 
from u will arrive at v which will cause (r,,d,) = (q¢,d +1). (Note that this packet 
will be accepted at v because in the previous state (r,,d,) > (q,d+1) as we have just 
shown and d < n’.) Also since u is the lowest ID neighbor at distance d from gq, v will 


set parent, = u and this will remain unchanged for the rest of the execution. J 


Theorem 8.3.8 T|L stabilizes to the stable tree behaviors for graph G in O(D) time, 
where D is the diameter of G. 


Proof: From Lemma 8.3.7 fj 


At this stage, we could apply the Global Correction Theorem to obtain a stabilizing 
version of J. However, there is an even simpler transformation because J is one-way 
checkable. Before we present the final version of the stabilizing spanning tree protocol, 


we first define what we mean by a one-way checkable protocol. 


8.4 One Way Predicates and Local Checking 


In Section 8.1, we claimed that every locally checkable protocol could be stabilized 
using the global reset protocol of Chapter 7. The proof sketch suggested that this 
could be achieved by periodically doing a local snapshot of each local subsystem and 


making a reset request if a violation is detected. 


However, in this section we will show that if a locally checkable protocol P is also 
what we call one-way checkable, then we can locally check P using a simpler and faster 
method than doing a local snapshot. In this method, each node periodically sends its 
entire state to each of its neighbors. We can call this local checking by periodic sending 


of state or simply periodic sending. The question that remains is: when is periodic 
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sending adequate to detect local violations in lieu of a local snapshot? The answer is 
that periodic checking is sufficient if each local predicate is what we call a separable 


local predicate. 


Intuitively, a separable local predicate can be separated into two one-way predi- 
cates, one for each direction of a link subsystem. Intuitively a one-way predicate for 
link (u,v) is a predicate that only involves the state of u, the state of the link (u,v), 
and the state of node v. Note that it does not depend on the the state of the link (v, wu). 
Formally: 


Definition 8.4.1 Consider any network automaton N with graph G. Let (u,v) be 
any edge in G. We say that Ou, 1s a one-way predicate for edge (u,v) of N if: 


© Ou» 1s a local predicate of N for edge (u,v). 


e For any two states s and s', suppose s satisfies Oy, and s|u = s'|u and s|v = s'|v 
and s|(u,v) = s'|(u,v). Then s’ satisfies Ou». 


For example, consider the second predicate of the spanning tree protocol listed in 
8.7. Recall that it essentially says that the estimate values in transit from v to u 
are always non-increasing. Clearly this is a one-way predicate. It is not hard to see 
that the first predicate (which involves only the state of one node) is also a one-way 


predicate. 


Definition 8.4.2 Consider any network automaton N with graph G. Let (u,v) be 
any edge in G. We say that L,,, 1s a separable local predicate for edge (u,v) of of N 
if there exist two one-way predicates O,, and Oy, (one for edge (u,v) and edge (v, wu) 


respectively) such that: 
Any state s of N satisfies Ly» iff 5 satisfies both Ou, and Ovu 


Once again it is easy to see that the spanning tree protocol has a link predicate 
set that consists of separable predicates. In this case, we say that the spanning tree 


protocol is one-way checkable. 


Definition 8.4.3 A network automaton N is one-way checkable for predicate L using 
link predicate set L tf: 
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e N is locally checkable for predicate L using link predicate set L. 


e Each Ly, € £ is a separable local predicate. 


We now claim the following fact informally. Any one-way checkable automaton can 
be locally checked by periodic sending. We simply add actions to send the state of a 


node periodically to each of its neighbors and actions to receive such packets. 


Suppose that in state s, node wu receives such a periodic packet from node v con- 
taining the value z. Then u checks whether (a, nil, nil, s|u) € Oyu; if this is not true u 


detects a local violation. 


Notice that local checking by periodic sending is not just simpler than doing a 
local snapshot but is also faster. The proof of Lemma 5.6.12 in Chapter 5 tells us 
that it takes 6 checking/correction phases for the local snapshot protocol to stabilize. 
Roughly, this means that the local snapshot protocol takes 12 link delays (where a link 
delay is the time to send a control packet from a node to its neighbor) to stabilize. 
By contrast, the periodic sending protocol takes 1 link delay to stabilize. Thus for 


one-way checkable protocols it always pays to use periodic sending for local checking. 


8.5 Simple Stabilizing Spanning Tree Protocol 


We have shown that the spanning tree protocol described in Figure 8.6 is locally check- 
able. Thus we can apply the Global Correction theorem to this protocol. This requires 
checking link predicates using local snapshots. However, since the spanning tree pro- 
tocol described in Figure 8.6 is also one-way checkable, we can can replace the local 
snapshot by periodic sending of state information. The resulting protocol is simpler 


and faster than a spanning tree protocol that uses local snapshots. 


If we look at the protocol in Figure 8.6, we see that each node sends (Announce, r, d) 
packets periodically containing the root and distance estimates at the node. But if we 
look at the predicates in Figure 8.7, Ol(u,v) involves only variables at u, and O2(u, v) 
only involves root and distance estimates at v. Thus we can rely on the (Announce,r, d) 
message for pertodic checking. This is shown in the code of Figure 8.8 which only 


describes the changes to the code of Figure 8.6 to convert it into a stabilizing protocol. 
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The first change is that we have applied message renaming to the original protocol 


and replaced all packet events with message events. 


The rule for one-way checking is as follows. If in state s, node u receives a periodic 
message from node v containing the value x, then u checks whether (2, nil, nil, s|u) € 
Ova; if this is not true wu detects a local violation. In our case, the (Announce,r, d) 
message does not contain the complete state x of node v, but only a projection of the 
state of node v that is sufficient to do checking. But (a, nil, nil, s|u) € O2(v,u), iff 
(r,d) < (ru[v],du[v]). Thus to check whether O2(v,u) holds it is sufficient to check 
whether (r,d) < (ru[v], dy[v]) and make a reset request if a violation is detected. Also 
notice that we do not need to check for the Ol(v,u) predicate because we have added 
a few lines of code that ensure that O1(v,w) will hold after the first Announce message 


received by v. 


Let T/ be the modified node automaton shown in Figure 8.8. Let R|L be the reset 
automaton for graph G as described in Chapter 7. Let 7’ be the automaton formed 
by composing T/ for each node u with RIL. 


Then we have the following theorem: 
Theorem 8.5.1 T’ stabilizes to the stable tree behaviors for graph G in O(n) time. 


Proof: (Sketch) We first show that J’ stabilizes to the behaviors of R(T(c)|L) in 
O(n) time. The operator R denotes message-renaming, as in the Global Correction 
Theorem. This part of the proof uses arguments similar to the ones used in the proof 
of the Global Correction Theorem except that the arguments are simpler because of 


the use of one-way predicates and one-way checking. 


Next, we know from Theorem 8.3.8 that T|L stabilizes to the stable tree behaviors 
of graph G in O(D) time. D is the diameter of G and is no greater than n, the 
number of nodes. But if T|L stabilizes to the behaviors in P in O(D) time, then so 
does R(T(c)|L). This is because the set of stable tree behaviors is invariant under the 
operations of message-renaming and scaling by constant factors. Finally, since behavior 
stabilization is transitive, we conclude that R(T (c)|L) stabilizes to the behaviors in P 


in O(D+n)=O(n) time. I 
Finally, we note that because T/ is a UIOA and RIL is a CIOA the modularity 


theorem assures us that we can replace R|L by its stabilizing implementation (Rt as 
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described in Chapter 7) without affecting the result. This is an elegant application of 
the modularity theorem because it means that we can begin with a stabilizing version of 
a UDL; next, as shown in Chapter 7 we can use the stabilizing UDL implementations to 
construct a stabilizing version of a reset protocol; and finally we can use the stabilizing 
version of the reset protocol to construct a stabilizing spanning tree protocol. In the 
process, we have two applications of the modularity theorem. We can go even further 
and change 7’ so that it offers a suitable interface to the token passing example of 
Chapter 6. If we do so, we could construct a stabilizing token passing protocol using 


the stabilizing spanning tree, once again relying on the modularity theorem. 


In any of these modular constructions, we can replace major pieces of the construc- 


tion by other modules that offer the same external behavior. 


8.6 Stabilizing topology update protocol 


The topology update task is to broadcast the list of incident operating edges at a node 
to all other network nodes. Thus the goal of the topology update task is to produce 
at each node wu a database listing the operating edges of each node that is reachable 


from uw. 


The following simple strategy is often used for topology maintenance. Each node 
maintains a sequence number. Whenever a change occurs, the incident nodes increment 
their sequence number and broadcast the new status of all their incident links in a 
link state packet ([Per83]). Any link state packet sent by node u contains node u’s 
sequence number. Whenever a node v receives a message purporting to be from u, v 
checks whether the sequence number on the message is larger (i.e., newer) than the 
sequence number of the last update v has stored from wu. If so, v stores this new update 
from u (after discarding any previous update it has stored from u) and broadcasts the 
update to all neighbors. Otherwise, the message is simply discarded as an outdated 


update. 


Now the sequence number field is finite. Even if we allocate 64 bits for sequence 
numbers, it is always possible (due to errors) for a link state packet with a maximum 


sequence number value to be present in the initial state of the network. 


In the ARPANET [MRR80] and DECNET [Per83] protocols, erroneous updates 


with the maximum number are removed by what we had earlier called “timer flushing”. 
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State 


requestbit, |v] boolean, true if a request is to be done later. 


All other variables remain unchanged. 


Modified Actions 


FREEMy,y (*message-renamed free action, replaces FREE,,, action*) 
Effects: free,|v] = true 


SENDMy,»(p) (*message-renamed output action to send estimate to neighbor v*) 
Preconditions: p is head of quewe,,[v] and free,,[v] = true 


Effects: free,|v] = false; Remove p from queue,,|v] 


RECEIVEMy,,( Announce, r, d) (*input action to receive estimate from neighbor v*) 
Effects: 


If (r, d) < (ru[v], du[v]) then (*received estimate is no greater than previous estimate”) 


Ifd<n! (*distance estimate not already at max value*) 
(ru[v], dulv]) := (7, d) (*update estimate from v*) 
Else requestbit,, = true (*if received estimate is greater make reset request later*) 
(ru[u], du[u]) := (u, —1); (*next two lines make O1(u, v) hold*) 


(Tu, dy, parent, ) := min{(ry|v], du[v] + 1,v),v € neighborSet,, } 


REQUEST, (*output action to request a reset”*) 
Preconditions: requestbit,, = true 
Effects: requestbit,, = false 


SIGNALy (*input action to receive a reset signal*) 
Effects: 
For all v 4 u € neighborSet,, do 
(ru[v], dy[v]) := (00, co) (*reset all estimates*) 


free, |v] := false 
(ru[u], dulu]) := (u, -1); 
(Tu, dy, parent, ) := (u,0,u) 


Figure 8.8: Spanning tree protocol. Modifications to the Code for a node w. 
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Each update (and hence the associated counter value) is assumed to remain in the 
network only for some limited “lifetime,” after which it is discarded. This prevents 
the problem because after its lifetime expires, an erroneous counter value is no longer 
present in the network. Once again, the disadvantage of timer flushing is that the 


timeout periods have to be high, which results in high stabilization times. 


Instead of using timer flushing, we can use global correction to create a faster 
stabilizing topology update protocol. This is because the protocol is easily seen to 
be locally checkable for the property L which states that the maximum value of any 
sequence number is not present in the network. Thus we can apply the Global Cor- 
rection theorem to ensure that maximum size sequence numbers are removed from the 
network. On receiving a signal, each node deletes the link state packet of all nodes 
other than itself, and resets its own sequence number to 0. A node that does not have 


a link state packet from u will accept any update sent by u as being newer. 


Thus, the number of bits allocated for the sequence number affects only the perfor- 
mance of the algorithm: errors that cause the sequence numbers to reach the maximum 


value incur a performance penalty — a network latency of O(n). 


A complete design of a stabilizing topology update protocol would also have to 
add a number of other actions as suggested in [Per83]. For instance, each node must 
periodically increment its sequence number and send its latest Link State Packets to 
all its neighbors. Basically, we suggest keeping the essential simplicity of [Per83] and 


only replace the need for global timers in [Per83] with global correction. 


8.7 Summary 


The major point of this chapter is to show that any locally checkable protocol can be 
stabilized using the global reset protocol developed in the last chapter. In general, this 
is done by having the leader of every every link subsystem do a periodic snapshot; if the 
leader detects a local violation, it makes a reset request. However, in the special case 
that the local predicates can be separated into one-way predicates, we can dispense 
with a snapshot and use periodic sending of state. Local Checking by periodic sending 


is simpler and faster than using a local snapshot. 


An important example of both techniques is furnished by the spanning tree pro- 


tocol described in Section 8.3. Both the spanning tree and topology update protocols 
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described are quite practical and could be used in real networks. 


The topology update protocol demonstrates a useful paradigm for designing stabi- 
lizing protocols. The essence of the idea is to design a stabilizing protocol assuming 
the use of unbounded counters. Next, we use global correction to convert this protocol 


into a more practical protocol that uses bounded counters. 


This chapter also demonstrates why the specification of a reset protocol must spec- 
ify the behavior in non-final signal intervals and not just in the final signal intervals. 
The proof of the Global Correction relies strongly on the mating relation between non- 
final intervals. The intuition is that the augmented automaton is doing local checking 
of the system during non-final intervals; if local checking does not eventually observe 
consistent behavior, the augmented automaton may keep making reset requests and 


the protocol may never stabilize. 
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Chapter 9 


Compiling Synchronous Protocols 


In the previous chapters we have shown how we can stabilize certain protocols by 
local checking and/or correction of the state of a node and its neighbors. We applied 
this technique to obtain stabilizing solutions to some important tasks such as mutual 
exclusion, network resets, and computing spanning trees. By contrast, this chapter 
provides a general technique for a special class of problems (known as non-interactive 
tasks), for many of which (e.g., Minimal Spanning Tree, Min Cost Flows, etc.) no 
locally-checkable implementation is known. In fact, for many of the problems solved 


in this chapter, no efficient stabilizing solution was known previously. 


In this chapter, we describe two compilers (Sections 9.5 and 9.6) that convert a 
deterministic synchronous protocol z into a stabilizing, asynchronous version of 7. In 
essence, we efficiently transform a solution for the most restrictive model (synchronous, 
fault-free networks) to a solution that works in a very permissive model (asynchronous 


networks with catastrophic faults that stop). 


Let T, be the time complexity of 7 and S, be the space complexity of 7. Let n be 
the number of nodes in the network. The first compiler produces a version of 7 that 
stabilizes in time O(n + T,) and has the same space complexity as 7. Thus the first 
compiler is extremely efficient if the time complexity of the original protocol 7 is at 
least O(n). The second compiler produces a version of 7 that stabilizes in time O(T, ) 
and has a space complexity of JT, -S,. Thus the second compiler is extremely efficient 


if the time complexity of the original protocol 7 is very small —i.e., T, = O(log(n)). 


These two compilers allow us to provide efficient stabilizing solutions for many 
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problems including minimum spanning trees, colorings, maximum flows, and maximal 


independent sets. 


Despite the apparent change of direction, the ideas in this chapter are closely re- 
lated to the ideas in the previous chapters. First, as we have said earlier, the ideas in 
this chapter extend the range of our general techniques. Our previous general tech- 
niques (Local, Tree and Global Correction) only apply to protocols that are locally 
checkable. Second, the compilers in this chapter are implemented using the the tech- 
niques developed earlier. The first compiler is based on a simplified form of local 
checking and correction that we call one-way checking and correction (see Chapter 8). 


The second compiler is based on local checking and global correction. 


When we first presented these results in [AV91], the second compiler, the Resynchronizer, 
was based on a complicated construction. In this chapter, we provide a simplified con- 
struction using the reset protocol described in Chapter 7. We do not have a complete 
proof of the simplified construction, but we will outline why we believe our simpli- 
fied construction is correct. Thus while our confidence in the Resynchronizer result is 
based on the original result in [AV91], we believe that the construction in this chapter 


offers the potential for a much simpler solution. 


This chapter is organized as follows. First, we describe how we model interactive 
protocols and synchronous protocols. Next, we summarize the major results of the 
chapter. Then we contrast the notion of distributed checking (that we use in this 
Chapter) with the independent local checking that we have used in previous chapter. 
Next we briefly describe the first Rollback compiler and then describe the simplified 
version of the Resynchronizer compiler. Finally, we outline extensions for randomized 


protocols. 


9.1 Non-interactive Protocols 


Non-interactive protocols form an important subclass of distributed algorithms. These 
are protocols whose correctness can specified by a relation (the I/O relation) between 
its input and output. For example, if the protocol has to compute a spanning tree 
then the output (the tree) should be a spanning tree of the input (the graph G). By 
contrast, mutual exclusion is an interactive task because (see Chapter 6) its correctness 


must be expressed in terms of sets of valid behaviors or sets of valid executions. 
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We define the notion of a non-interactive protocol more formally using a slight 
variant of the network model introduced in Chapter 5. Recall that in Chapter 5, we 
modelled a network using a topology graph G = (V, F,1), where V is a set of nodes, E 
is a set of directed edges and / is a leader function. For this chapter we will augment 
the notion of a topology graph to have an extra component I that represents the input. 
We will define an augmented topology graph G = (V,E,1,I) where V,E and / are as 
before. I, however, is a vector of inputs such that for any node u, J, represents the 
local input at node wu. (In previous chapters the only input to a node automaton for 


node u was the list of neighbors of u and, in the case of a tree topology, the parent of 


We start by fixing an input domain 7 and an output domain O. 


Definition 9.1.1 An augmented topology graph G = (V,E,1,I) is a topology graph 
(V,E,1) together with an input specifier I, where I is vector of local inputs (I) and 
L,ET,Vuev. 


Informally, this amounts to modelling the input to the non-interactive protocol as 
part of the local code of each node (which cannot be corrupted) and not as part of 
the state of each node (which can be corrupted). Clearly no non-interactive protocol 
can hope to produce correct output from corrupted input! However, more generally, 
the input to the protocol could be the output of another stabilizing protocol. Thus in 
Chapter 5, we assumed the list of neighbors of each node u was part of the code at 
node u. But in many real networks, the list of neighbors of a node would be provided 
as the output of a stabilizing Data Link protocol. In this chapter, we will often refer 


to an augmented topology graph as an augmented graph. 


Once again, we will model the protocol using node automata WN, (one for each node 


u in G), and unit capacity data links C,,,, (one for each edge (u,v) in G). 


Definition 9.1.2 An automaton for augmented graph G = (V, E,1,I) is an automaton 
consisting of an IOA for each u € V together with a UDL Cy for each (u,v) € E. 


Refer to Chapter 5 for a definition of the UDL C,,,. Recall that for any state s of 


the automaton we let s|u denote the state of the IOA for node wu. For an augmented 
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graph, we also let (X,,) denote a vector consisting of an element X,, for each node u 


in the graph. 


For non-interactive protocols we will also assume that there is some local output 
function O, of the state of each N,,, such that O,,(s) provides the local output of N,, in 
state s. Let N be the network automaton formed by the composition of the node and 
channel automata as described in Chapter 5. Using the O, we can define the output 
function O to be a function of the state s of the network automaton N such that O(s) 
produces a vector of local outputs that is identical to the outputs produced by each 


O, when applied to s|u. Thus: 


Definition 9.1.3 Let N be an automaton for augmented graph G = (V,E,1,I). We 
say that O is an output function for N if O is a vector of functions (O.), such that 
for every state s of N and for every node u € V, Ou(s|u) € O. We will also abuse 
notation by defining O(s) = (O.(s|u)), Vu € V. 


Traditionally, the correctness of a non-interactive protocol N is described using an 
input-output relation R. In the traditional definitions, NV is allowed to be an ordinary 
IOA (i.e., we are allowed to specify initial states). Next, the correct executions of N 
are those executions a of N in which there is some t-suffix y of a such that for all 
states sin y, (J,O(s)) € R. In other words, in all executions of the protocol the output 


must eventually “settle” to a value that satisfies the I/O relation. Thus: 


Definition 9.1.4 Let N be an automaton for augmented graph G = (V,E,1,I). An 
I/O relation for N is a set of tuples (I',O') where each I' is a vector (I'), I) € I and 
each O' is a vector (O!,),O1,€ O. 


Finally we define a non-interactive protocol to consist of two components: an 


automaton and an output function. 


Definition 9.1.5 A non-interactive protocol for augmented graph G is a tuple (N,O) 


where N is an automaton for augmented graph G and O is an output function for N. 


For stabilization, we will keep the the correctness definition exactly as in the tra- 
ditional definitions except that N will typically be a UIOA. 
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Definition 9.1.6 Let P = (N,O) be a non-interactive protocol for augmented graph 
G =(V,E,1,I) and let R be an I/O relation for N. Let C be the set of executions a of 
N such that for all states s in a, (I,O(s)) € R. We say that non-interactive protocol 
P stabilizes to I/O relation R in time t if N stabilizes to the executions in C in time 


t. 


Thus this definition is a special case of the general definition of “stabilizes to the 


executions of” given in Chapter 3. 


We now formally define two complexity measures that we use to evaluate stabilizing 
non-interactive protocols: the first measures the worst-case time for the protocol to 


stabilize, and the second measures the worst-case amount of space used by any node. 


We define the stabilization time of a non-interactive protocol P with respect to I/O 
relation R to be the infimum of all t such that P stabilizes to R in time t. 


The space complezity of P = (N,O) is the worst case across all nodes u of the size 
of the set {s|u: 5 € states(N)} 


9.2 Synchronous Protocols 


We model a deterministic synchronous protocol 7 as follows. The network topology is 
again specified by an augmented graph G = (V, E,1,I), where I, once again represents 
the input to node u. The protocol is synchronous because it works in rounds numbered 
from 0 to T,, where T, is some finite, known bound on the running time. The channels 
are no longer UDL’s but simply obey the property that any packet sent on a channel at 
the start of a round is delivered by the end of the round. Similarly, the node protocol 
at each node wu is no longer modelled by a node automaton but instead by a 4-tuple 
(s0, Fu, Mu,Ou), where s° is an initial state, F,, is a state transition function, M, is a 
message generation function, and O, is the output function. All these correspond to 


the standard notions for such protocols but we explain them briefly below. 


The state s of the synchronous protocol consists of the state s|u of each node in G. 
An ezecution of the synchronous protocol is generated as follows. At the start of the 
first round, each node u is placed in the initial state s® and all channels are empty. 


At the start of each round, a node sends a packet to each neighbor. The contents of 
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the packet are determined by the message generation function M,, that takes as input 
the state of u at the start of the round. At the end of each round, a node changes its 
state using the state transition function F,, that takes as input the messages received 


by u in the round as well as the state at the start of the round. 


The output of 7 in any execution is determined by the value of O,(s|w) for every 
node u, where s|u is the state of node wu at the end of round T,. Exactly as for non- 
interactive protocols, we define an output function O to be a function of the state s 
of x such that O(s) produces a vector of local outputs that is identical to the outputs 


produced by each O, when applied to s|u. 


Exactly as for non-interactive protocols, we define the correctness of synchronous 
protocols using an I/O relation R. The synchronous protocol z is said to be correct if 
all runs of a end in a state s such that (J,O(s)) € R. 


The time complexity of a synchronous protocol 7 is the number of rounds, T,. The 


space complexity of 7 is the worst case across all nodes u of the number of states of wu. 


Let 7 be a synchronous protocol with time complexity T, and Input-Output relation 
R. We also introduce the notion of a checker for 7. A synchronous protocol y is said 


to be a checker for 7x if it satisfies the following property. 


Suppose each node wu in y is given as input both the input of z (i.e., J) as well as 
a value O’ that purports to be the output of 7 on input J. Informally, y must detect 
(at some node) if the purported output O’ of 7 could have been produced in some run 
of 7 on input J. Thus y has a boolean output at every node. If any node outputs the 
value false, then it must be that O’ could never have been produced by 7; if all all 


nodes output the value true, then it must be that O’ could have been produced by 7z. 


Note that our definition only allows deterministic synchronous protocols whose 
output is a function of its input. Thus for such protocols 7, 7 has a trivial checker that 
simply runs 7 again and compares the output O,(s) at each node with the purported 
output O!,. The checker outputs true at node u iff O.(s) = O},. 


9.3. Results 


Recall the definitions of stabilization time and space complexity for a non-interactive 


protocol P and the definitions of time and space complexity for a synchronous protocol 
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Stab. 
Method Space 
Time 


Rollback Tx T+ Sx 


Resynchronizer | T,, +n Sx 


Figure 9.1: Comparison of the complexity of our compilers 


aw. Recall that n is the number of nodes in the network. 


Our solutions are in the form of compilers that can compile synchronous protocols 
into stabilizing versions that have the same I/O relation when run on an asynchronous 


network. The performance of our compilers is summarized in Figure 9.1 


Our simplest compiler is the Rollback compiler that takes a synchronous protocol 
mw as input and produces a stabilizing, non-interactive version of 7. This is expressed 


as the following theorem: 


Theorem 9.3.1 Rollback Compiler: Consider any synchronous protocol 7 for aug- 
mented graph G with I/O relation R, time complezity T, and space complezity S,. 


Then there is a corresponding non-interactive protocol P =(N,O) such that: 


e N is a UIOA. 
e The space complexity of P is O(T, - S;). 


e The stabilization time of P with respect to R is O(T,). 


The main contribution of this chapter is a second compiler called a Resynchronizer. 
In its simplest form, the Resynchronizer takes a deterministic synchronous protocol 7 


as input and produces a stabilizing non-interactive version of 7. 


Theorem 9.3.2 Resynchronizer Compiler: Consider any synchronous protocol 
a for augmented graph G with with I/O relation R, time complezity T, and space 
complexity S,. Then there is a corresponding non-interactive protocol P = (N,O) 
such that: 
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e N is a UIOA. 
e The space complexity of P is O(S,). 


e The stabilization time of P with respect to R is O(T, +n). 


Our original proof of the Resynchronizer Compiler Theorem was based on a com- 
plicated construction given in |[AV91]. We conjecture that the simplified construction 


in Section 9.6 also provides a proof of the same theorem. 


We will also see in Section 9.7.2 how to use the Resynchronizer to compile random- 
ized, synchronous protocols (of course, this would entail generalizing our model of a 


synchronous protocol to allow coin-tosses at each node.) 


Note that, when compared to the Resynchronizer, the Rollback construction re- 
moves the additive factor of n in the stabilization time but increases the space by a 
factor of T,. Thus Rollback is useful only for “fast” protocols that have T, <n. The 


two compilers can be used to efficiently stabilize several non-interactive tasks. 


Some sample results obtained by applying the Resynchronizer are as follows. For 
the problems of computing a spanning tree and single source shortest paths we achieve 
O(n) stabilization time and O(log n) space. This is comparable to the results achieved 
in Chapter 7 and the best previous results. For the problem of computing a minimal 
spanning tree [GHS83] we achieve O(n) stabilization time (which is as good as the 
time of the best non-stabilizing synchronous protocol) using O(log) space. For the 
problem of computing a maximum flow [Gol85, GT88] we achieve O(n?) stabilization 


time, which is as good as the time of the best non-stabilizing synchronous protocol. 


The Rollback compiler gives good results when applied to symmetry breaking prob- 
lems such as the problems of computing a Maximum Independent Set [AGLP89], A+1 
Coloring in sparse networks [GPS87], and A? Coloring in general networks [Lin87]. For 
instance, for A + 1 Coloring in sparse networks we achieve log” n for both measures. 


This is much better than any previous results. 


9.4 Distributed Program Checking 


A deterministic sequential algorithm can make itself stabilizing by periodically saving 


its output and running itself again; when it finishes it can check its output. For 
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sequential algorithms, this is ugly and unnatural — after all, we want the computer to 


move on and run other programs! 


However, as we have said before, periodic checking is quite natural for distributed 
computing. Once we have accepted the inevitable periodic cost of stabilization we can 
ask: why not run a checking process to check the algorithm periodically?. If the check 
reveals a problem, we restart the main algorithm. In the previous chapters we have 
already seen how we can do this in some cases if each link subsystem independently 
does local checking. However, that method (which we can call independent checking 


of link subsystems) is limited to locally checkable protocols. 
One approach to check a distributed program introduced by Katz and Perry [KP90] 


is to collect all information at a single “leader” node, thus reducing distributed checking 
to centralized checking. However, the space and time complexities of this method are 


quite large because of the bottleneck at the “leader” node. 


In this chapter, we will introduce a form of distributed checking that is neither 
the independent local checking of the previous chapters nor the centralized checking 
introduced by Katz and Perry. In many cases, we can improve performance by doing 


distributed program checking. 


Such distributed program checking clearly requires coordination of all the network 
nodes. For example, we need to ensure that a node not move to a new phase (whether 
it be checking or executing) before every other node in the network has completed this 
phase. The main difficulty is to implement this coordination in a stabilizing fashion. 
We will describe two such implementations — the Rollback protocol in Section 9.5 and 


the Resynchronizer in Section 9.6. 


9.5 Rollback Compiler 


There is a naive implementation of distributed checking that requires a large amount 
of storage and bandwidth and works only for deterministic protocols. In the naive 
method, every node keeps a log of every state transition it has taken to reach its 
current state. If each node constantly sends its current log to all neighbors, every 
node can check and correct every transition it has made in the past. Since the inputs 
are always correct, eventually all transitions are corrected. This method works only 


because it is possible to check transitions in a stabilizing fashion. 
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Clearly, for an arbitrary asynchronous protocol the logs can grow very large. How- 
ever, if the asynchronous protocol is simulating an underlying synchronous protocol 7 
then the size of the log can be reduced to the time complexity T, of 7. This idea is 
implemented in the dynamic synchronizer of |AS88]. However, in [AS88] the logs are 
only used to avoid unnecessary recomputation after an input change, and hence are 
not periodically checked. By adding the periodic checking of logs, we obtain a Modified 
Dynamic Synchronizer that we call Rollback. 


The disadvantage of the Rollback method is that it blows up the space utilization 
and periodic bandwidth by a factor of T,. This is not a problem for protocols with 
small time complexity — i.e., those for which T, < n. Thus the naive method leads 
to efficient solutions for such problems as coloring and MIS. However, for protocols in 
which 7, >, Rollback is a poor choice. 


Notice that the Rollback compiler combines the use of logs with a simplified form of 
local checking and correction. The Rollback compiler does local checking by periodic 
sending of state (as opposed to doing a local snapshot). This is because the Rollback 
compiler is one-way checkable (see Chapter 8 for a definition of a one-way checkable 
protocol). Also, the Rollback compiler does local correction by simply correcting the 
local state of a node when periodic sending detects a problem (as opposed to doing a 
local reset). If local correction can be performed in this simple manner, we will call the 
protocol one-way correctable. Chapter 10 discusses one-way correctability in a little 


more detail. 


9.6 Simplified Resynchronizer Compiler 


In the previous section, we saw how Rollback did distributed checking by maintaining 
a log of its computation. Consider a deterministic protocol 7. Clearly, we can avoid 
the need for a log if we could simply re-execute 7. However, this constant re-execution 
requires coordination among the network nodes which must be implemented in a sta- 
bilizing fashion. In general, the Resynchronizer avoids a log by constantly re-executing 
a checker x for 7. We introduce the basic idea by assuming that 7 is deterministic 


and that 7 is its own checker. We return to separate checking later. 


The Resynchronizer can be thought of as a stabilizing version of a synchronizer 


[Awe85]. Any synchronous protocol can be simulated asynchronously by using a pulse 
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number at each network node. Let us call a node synchronized if its pulse number 
differs by at most 1 from any of its neighbors. In ordinary synchronizers, every node is 
initially synchronized by setting the pulse numbers of all nodes to 0. Thereafter, a node 
increments its pulse number from p to p+ 1 only after all its neighbors have reached 
pulse p, thereby maintaining synchronization. If each node executes the corresponding 
step of the synchronous protocol at pulse p just before incrementing to p+ 1, then 
the asynchronous protocol has the same I/O relation as the underlying synchronous 


protocol. 


Since a stabilizing synchronizer cannot rely on correct initialization, we introduce 
a Termination_Detection phase. This phase will two do things. First, this phase 
ensures that each node has finished executing the synchronous protocol. Second, once 
each node has finished executing the synchronous protocol, a reset (see Chapter 7) 
is initiated. Once all nodes get a signal (see Chapter 7) all nodes reset their pulse 


numbers to 0 and the cycle continues. 


Thus the Resynchronizer can be considered to be an application of global correction 
to the simplest synchronizer protocol described in [Awe85]. Our original construction 
and proof |AV91] relied on a special-purpose reset protocol that was specially crafted 
to work with the synchronizer protocol. In this chapter, we will describe a simplified 
version of the construction that uses the general purpose network reset protocol of 
Chapter 7. While we do not have a complete proof of the simplified construction, 
we believe the construction in this chapter provides the basis for a simpler and more 


elegant solution. In what follows, we only describe the simplified construction. 


When pulse, € [0,7,| we say that node wu is in the Execute phase. In the Execute 
phase node u simply executes the normal synchronizer algorithm described earlier. It 
also implements the underlying synchronous protocol, starting from initialization at 
pulse 0 followed by writing its output at pulse T,. Let D be an upper bound on the 


diameter of the network.! We assume that D <n. 


We denote by Maz = T, + D the maximum value of pulse,. When pulse, € 
[T, + 1, Maz] we say that node wu is in the Termination Detection phase. Suppose 
that all nodes are synchronized when some node exits the Execute phase. Then the 


Termination_Detection phase ensures that every node has had a chance to correct its 


1Unfortunately our protocol needs an upper bound on the diameter of the network. This is a liability 


of the protocol. 
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output before the pulse number of some node wraps around to 0. If the nodes are 
synchronized by the start of this phase, a node need only wait D dummy pulses to 


make sure that all other nodes have reached the Termination_Detection phase. 


If pulse, reaches Maz, node u makes a reset request and waits to get a signal. This 
potentially destroys synchronization. However, we can rely on the properties of the 
reset protocol to deliver a signal at all nodes in a consistent way. When node wu receives 
a signal it sets its pulse number to 0 and the cycle continues. Notice that all nodes 


constantly reexecute the underlying synchronous protocol. 


The Resynch compiler must deal with arbitrary pulse numbers and arbitrary mes- 
sages on links in the initial state. Second, it must cope with the fact that the pulse 
numbers are finite and hence have to wrap around. Recall that one of our motivations 
for studying stabilization was to make real network protocols more fault-tolerant. Any 


real counter implementation is bounded. 


In our description and in the code below we only describe the Resynchronizer as a 
tool that can be used to compile a deterministic protocol by re-executing it. It is easy 


to extend these ideas slightly to add a separate checking phase to the Resynchronizer. 


9.6.1 Resynchronizer Code 


We described how to reduce the problem of stabilizing the output of a synchronous 
protocol 7 to the problem of building a stabilizing synchronizer that constantly re- 
executes pulse numbers in the execute region. Thus when presenting the code, for 
simplicity we ignore the details of executing 7; instead we concentrate on the major 
task of synchronizing pulse numbers. To actually execute 7, we need to supplement 


the code as follows: 


e Additional state variables are added at every node u to keep track of the state 
of 7. Also, we assume that every pulse message with pulse p carries the state of 


m (at the sending node) on pulse number p and p — 1. 


e Whenever pulse, reaches p and p is in the Execute phase, the synchronous pro- 
tocol 7 is executed at node u. (Sometimes the code will cause the pulse number 
at node u to jump from say p to p+ 2. In that case, the synchronous protocol 


must be executed at pulse p+ 1 and pulse p + 2.) 
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e Whenever pulse, reaches T,, node u corrects its output to be the local output of 


TT. 


The protocol is formally presented below in Figures 9.2 and 9.3. Every node u 
keeps the variables described in Figure 9.2. 


We assume that each node is a user of the reset protocol R|L described in Chapter 
7. As in Chapter 7, we assume that R|L has an interface to make reset requests 
and accept reset signals as shown in Figure 7.2 in Chapter 7. Recall also that the 
reset subsystem also has an output action FREEM that tells the user (in this case the 
spanning tree protocol) when it can send a new user message. Of course, the similarity 
to the FREE action of a UDL is no coincidence. Just as the user of a UDL needs to 
keep a free variable to record whether the UDL is free, each node (exactly as in the 
spanning tree protocol of Chapter 8) will keep a variable free for each neighbor. In 
fact, the entire protocol bears a strong resemblance to the spanning tree protocol of 


Chapter 8 although it performs an entirely different function. 


Every node wu periodically sends its own pulse number to all neighbors. using a 
“Pulse” message. The Pulse message is encoded as a tuple (Pulse,p). When a node 
v receives an Pulse message from u, v compares the pulse in the message (say p) to 
the previous pulse estimate stored from u (pulse,|u]). During normal synchronizer 
operation the following local predicate always holds: pulse,|u] <p and p < pulse, +1. 
(Intuitively, the pulse estimates sent by u to v are non-decreasing and are never more 
than one higher than the current pulse number of v) If v realizes that the local predicate 
has been violated, v makes a reset request. If this is not the case, v stores the latest 
estimate it has received from u; v also updates its own estimate as the minimum of all 


neighbor estimates. 


Once node uw finds that Pulse, = Maz, it makes a reset request. Once node wu gets 


a signal, node u resumes normal synchronizer operation. 


We assume that the topology of the network is specified by an augmented graph 
G. Let R|L be the reset protocol described in Chapter 7. Let 55, be the automaton 
shown in in Figure 9.3. Let S be the composition of the automata node S%S,, for every 


node in G with RIL. 
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Variables 


e pulse,: Highest pulse which was performed correctly until now. Its domain is restricted 
to lie in {0.. Maz}. Recall that Mar = T,,+ D, where D is an upper bound on the network 


diameter. 


e pulse,|v|: Node u’s estimate of the current pulse number at node v. In normal syn- 
chronizer, the estimate is always a lower bound on the true value. Its domain is also 
restricted to lie in {0..Maz}. 


e free,|v|: Boolean, which if true indicates that link to v is free to accept new messages. 


e requestbit, |v]: Boolean, which is set to true to remember that node u should make a 


reset request in the future. 


Figure 9.2: Declarations of variables at node u used by Resynchronizer code. 


9.6.2 Proof Sketch for the Simplified Resynchronizer Construc- 


tion. 


We outline an argument that explains why we believe the simplified construction is 


correct. We emphasize that this is an intuitive argument that needs further polishing. 


Just as in the spanning tree protocol of Chapter 8, the normal operation of the 
synchronizer can be defined by two one-way predicates S1(u,v) and S2(u,v) shown in 
Figure 9.4. The first states that a node’s pulse number is one higher than the smallest 
estimate it has from any neighbor). The next states that the pulse estimates sent by 
node u to neighbor v are strictly non-decreasing. However, any estimate sent by u can 
never be more than one higher than the current pulse number of v. If we compare the 
local predicates of Figure 8.7 with the local predicates of our spanning tree protocol 


(Figure 9.4) we notice a remarkable similarity between two different protocols. 


If these two local predicates hold for all edges (u,v) we can show that all nodes 
are synchronized - i.e., the pulse number of each node differs by at most 1 from any of 
its neighbors. This is sufficient to ensure that the pulse numbers will eventually grow 


until some node reaches the maximum value. 


Unlike the spanning tree protocol, however, the Resynchronizer protocol keeps 


making reset requests and there will never be final signal intervals in any execution 


251 


RECEIVEMy,,( Pulse, p) (*receive pulse message from neighbor v*) 
If (pulse, |v] < p) and (p < pulse, + 1) then 
pulse, |v] := p 
If min{pulse,|v],v € neighborSet,} + 1 < Maz then 
pulse, := min{pulse, |v], v € neighborSet,} +1 
Else pulse, := Maz 


Else requestbit, = true 


SENDM,,y( Pulse, p) (*output action to send pulse to neighbor v*) 
Preconditions: p = pulse, and free,|v| = true 
Effects: free,,|v] = false 


REQUEST, (*output action to request a reset”*) 
Preconditions: requestbit,, = true or pulse, = Maz 
Effects: None 


SIGNALy (*input action to receive a reset signal*) 
Effects: 
pulse, := 0 
For all neighbors v of u do 
pulse, [v] := 0 
free, |v] := false 
FREEMy,y (*receive free signal from link to neighbor v*) 
free, |v] := true 


Figure 9.3: Resynchronizer Code for any node u 
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of the protocol. Thus we need to make heavy use of the causality property of reset 
behaviors. For example, we can show that within O(n) time of any suffix of an exe- 
cution, some node must make a reset request. This is proved by contradiction. If not, 
in linear time all signals must stop and in constant time after this all local predicates 
must hold. (The second part follows because the code makes a reset request if it de- 
tects a violation of S2(u,v). Also, the receipt of the first pulse message at u will make 
S1(u,*) hold.) Thus by normal synchronizer operation, the node with the lowest pulse 
number will increase its pulse number by 1 in constant time; thus in O( Maz) = O(n) 
time, some node will have reached the maximum pulse number and hence will make a 


reset request. 


The overall argument consists of showing that there is some ¢-suffix suffix y of an 


execution of the Resynchronizer protocol, where ¢ = O(n) and such that: 


1. All nodes receive a signal event in linear time after the start of +. 


2. In linear time after all nodes reach pulse number 0, some node reaches pulse 


number Maz and all nodes reach pulse number T, = Maz — D 


As in the proof of the Global Correction Theorem, we can argue that in linear 
time, all reset requests that are caused by local predicate violations will disappear. 
Thus we choose y such that reset requests in y are only caused by nodes reaching the 


maximum pulse number Maz. 


The intuition behind the first part is that since some node makes a reset request 
within O(n) time of 7, within O(n) time in y all nodes will receive a signal. Notice 


that when a node gets a signal, the node resets its pulse number to 0. 


The intuition behind the second part is as follows. After all nodes have reached 
pulse number 0, we can again argue that some node makes a reset request in linear 
time. By our choice of 7, a reset request is only caused by some node, say wu, reaching 
pulse number Maz. But since u was previously at pulse number 0 in 7, u’s pulse 
number must have grown from 0 to Maz in y. We then argue that all neighbors of u 
have grown from 0 to Maz — 1 in 7, and finally that all nodes have grown from 0 to 


Maz— Din vy. 
Finally, if all nodes grow from 0 to Maz — D in y, we argue that by the end of y 


all nodes have correct output. Intuitively, this is because each node begins executing 


the synchronous protocol at pulse 0 and corrects its output at pulse 7, = Maz — D. 
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S1(u,v): 
pulse, = min{pulse, |v], v € neighborSet, } +1 


$2(u,v): 
If there is an (Pulse, p) message in transit from u to v then 
pulse, [u] < p< pulse, and p< pulse, + 1 
Else pulse, [u] < pulse, 


Figure 9.4: Normal Synchronizer: Local Predicates for edge (u, v). 


9.7 Extensions 


9.7.1 Better Synchronous Checkers for Deterministic Proto- 


cols 


Suppose we limit ourselves to stabilizing protocols 7 that execute a distinct checking 
protocol x even after reaching a legal state. Let us informally define the stabilization 
bandwidth of 7 to be the worst case message complexity per link of the checking 
protocol. Clearly the checking protocol must be executed at least once every T time 
units — where T is the stabilization time — even after the protocol has stabilized. 
Hence this is really a bandwidth cost. For example, the stabilization bandwidth of 
the protocol in [KP90] is the worst case message complexity per link to do a snapshot, 


which is Q(m), where m is the number of links. 


If we check a deterministic protocol 7 by re-executing 7, we have to pay a high 
price (T,,) in stabilization bandwidth. Suppose instead, we can find a checker y for 7 
that has T, = 1 —i.e., can check 7 in a single pulse. Then after the execution phase 
we can add a single check pulse number T,,. When a node reaches pulse T,, it stays 
at T., executing the checking protocol until it detects a problem; if it does detect a 
problem it drops back to pulse 0. By avoiding multiple pulses for checking, we remove 
the need for costly a termination detection and reset operation in the checking phase. 


The stabilization bandwidth drops to O(S;,). 
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There are a number of tasks that we can check in a single pulse. These include 
the classical problems of shortest paths, topology update, leader election, computing a 
spanning tree, and computing a maximum flow. Because the first four tasks are com- 


monly used in real networks, it is important to improve their stabilization bandwidth. 


We do so by what by essentially introducing a new form of local checking which 
we will call synchronous local checking. The idea of doing local checking was already 
introduced in previous chapters. In the previous chapters, we described asynchronous 
protocols which continuously checked the state of every link subsystem. Note that 
state of of every link subsystem includes the state of the channels as well as that 
of the nodes. However, to build an efficient synchronous checker, all we have to do 
is to locally check a synchronous protocol that has terminated. Hence the checking 
procedures are much simpler. Note that for a synchronous protocol, the state of every 


link subsystem consists only of the state of two nodes. 


Consider the problem of checking the shortest cost paths from a given source to 
every other node. The key is the ability to check, in a single pulse, the shortest 
cost distances from a given source to every other node. Let s denote the source 
and D; denote the distance node 7 has recorded to s; let B;; denote the cost of a 
link from Node i to Node j, and N(i) the set of neighbors of Node i. Then in a 
checking pulse a) s checks that D, = 0 b) Nodes other than s check that D; > 0 and 
D; = Minjen()(Dj + Bij). It is quite simple to prove that if any of the distances are 


wrong some node will detect this after checking. 


By adding an extra distance label to the other tasks, the other tasks can also be 
checked in one pulse, using the same trick. We can do this, for instance, in leader 
election by adding the distance to the leader, and in maximum flow by adding the 


distance to the source in the residual graph. 


In general, given our application, the design of faster checkers for synchronous 
protocols becomes an interesting and practical problem. For instance, we can we 
check a minimum spanning tree in fewer pulses than it would take to compute the 
tree from scratch while using only small storage? There are a number of similar open 


problems that arise from our work. 
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9.7.2 Randomized Protocols 


In all our models, both of asynchronous and synchronous protocols we have disallowed 
the possibility of nodes tossing coins. Thus our formal models preclude the description 
of randomized protocols. In a randomized protocol, nodes are given the ability to toss 
coins. The definitions of correctness now become probabilistic and it is much harder 
to give careful definitions of stabilization. However, we will assume that the reader 
has an intuitive understanding of randomized protocols to understand the following 


informal discussion. 


First, for randomized protocols, we will assume that any supposedly random bits 
in the initial state of a node can be arbitrarily corrupted and hence non-random. 


However, subsequent coin-tosses will produce truly random bits. 


Next, note that we need a separate checker to compile a randomized protocol. 
This is because re-executing the original protocol can lead to a different output, and 
cause the checker to detect an error when there was none. Saving the original random 
bits in the state does not help either as these bits could be corrupted (see previous 
assumption). Further, this checker must be oblivious: it must not depend on the 
correctness of the supposedly random bits currently in the state. It appears that a 
stabilizing algorithm that uses a randomized checker needs an infinite supply of random 


bits since it cannot rely on the old random bits at any stage. 


A simple example of compiling a randomized protocol is furnished by the problem 
of electing a leader in an anonymous network - i.e., a network in which nodes do 
not have any unique IDs.? Clearly we need randomization to break symmetry. To 
construct a stabilizing protocol for this task, we demonstrate a synchronous protocol 


for execution together with an oblivious synchronous checker. 


Recall that we use D to denote an upper bound on the diameter of the network. 
In the execution protocol, each node picks a random ID uniformly and independently 
in a space of 1..X. In the next D pulses, a node considers itself as the leader if it finds 
out that its ID is the largest in the network. In the checking protocol, each node z that 
considers itself a leader picks a new random value ¢; in the space 1..X and broadcasts 
t; during the next D pulses. At the end of D pulses, a node detects an error if it has 


received either no values or it has received more than one value. While both checking 


2If nodes have unique IDs we must assume that the IDs are protected; dropping this assumption 


makes the system more fault tolerant. 
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and execution can fail, by picking X to be a polynomial (of sufficiently high degree) in 
the number of nodes, we can ensure that a correct output will be produced in constant 
expected number of phases. Since each phase takes O(D) the ezpected stabilization 
time is O(D). 


A more efficient protocol for this purpose (that works in time proportional to the 
actual diameter as opposed to a bound on the diameter) is given in [DIM91b]. However, 
our solution seems to be simpler. As in the case of deterministic protocols, checking 


randomized synchronous protocols seems an interesting research area. 


9.8 Summary and open problems 


The main result of this chapter the Resynchronizer, is a compiler that transforms any 
synchronous protocol into a stabilizing version for dynamic asynchronous networks. 
The transformation adds O(n + D) overhead to the time complexity of the protocol, 
where D is a bound on the diameter of the final network. Clearly D and n can be 
much larger than the actual diameter of the final network. A natural open problem is 
to obtain a compiler whose time overhead only depends on the actual diameter of the 


final network. 


When the results in this chapter were first presented [AV91], the Resynchronizer 
compiler used a much more complicated construction (which could be regarded as a 
special reset protocol optimized for the case of synchronizer operation). The transfor- 
mation in [AV91]| added only O(D) overhead to the time complexity of the protocol, 
removing the factor of n which arises from the use of the reset protocol described in 
Chapter 7. However, for many real networks the only meaningful bound on the diame- 
ter of the network is the number of nodes. Thus this does not seem to be a meaningful 
distinction in practice. Also, the construction in this chapter is much simpler than the 


original. However, a careful proof of the simplified construction is needed. 
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Chapter 10 


Conclusions and Open Questions 


This thesis investigates the power and applicability of local checking and correction for 
the design of stabilizing network protocols. The thesis provides a rigorous theoretical 
foundation for these concepts. However, the emphasis is on using these concepts to 


devise novel fault-tolerant protocols for real networks. 


We have confined the notion of “local” to link subsystems consisting of a pair of 
neighboring nodes and the two unidirectional links between them.’ We have defined 
a protocol to be locally checkable for a good property L if two conditions hold. First, 
there is a local property for each link subsystem (formalized by a predicate of the link 
subsystem) such that property L holds if the local properties of all link subsystems 
hold. Second, each local property is closed - i.e, once the local property is true, it 


remains true. 


We have also defined a protocol to be locally correctable to property L if the protocol 
is locally checkable for LZ and there is a way to locally correct each node with respect 
to a link subsystem (formalized by the existence of a local reset function) such that the 
global property L becomes true. The index contains pointers to the formal definitions 


of these concepts. 


This chapter summarizes and evaluates the major contributions of the thesis. The 


chapter ends with a list of open questions. 


1However, in the list of open questions we suggest that this notion of locality can be usefully 


generalized. 


258 


10.1 Contributions 


The major contribution of this thesis is the construction of general techniques for de- 
signing stabilizing protocols. All our techniques revolve around the concepts of local 
checking and correction. We use the general techniques to design new stabilizing pro- 
tocols (many of which are practical) and to understand existing stabilizing protocols. 
In the process of formalizing these techniques, we were also led to invent new defini- 
tions of stabilization, a simple modularity theorem, and a new Data Link model. We 
believe this modelling effort is useful in its own right. Thus we divide the contributions 


of this thesis into five categories: 


e General techniques for constructing stabilizing protocols. 


e New or improved stabilizing solutions to specific problems. 


The Modularity Theorem 


Modelling of self-stabilization. 


e Better understanding of existing work in self-stabilization. 


We describe these contributions in more detail in the following subsections. 


10.1.1 General Techniques 


The thesis contains four general techniques and a powerful heuristic. The four general 
methods are all organized around the theme of local checking and form a natural 


progression of ideas. The methods are: 


e Local Correction: The Local Correction Theorem (Theorem 5.4.3) of Chapter 
5 shows that every locally checkable and correctable protocol can be stabilized in 
time proportional to the height of the underlying partial order. This is achieved 
by adding actions to do independent checking and resetting of each link subsystem 


The Local Correction Theorem formalizes a useful design technique for building 


stabilizing protocols. First, we add some (hopefully small) state to the protocol 
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to make it locally checkable. Next, we look for a local reset function that can be 
used for local correction. In some cases, it is possible to construct a local reset 
function by combining several actions of the original protocol. In Chapter 7 we 
conjectured that this (i.e., the construction of a local reset function) is possible 
for many existing protocols that work in dynamic networks. The stabilizing reset 


protocol of Chapter 7 is an example of this design technique. 


e Tree Correction: The Tree Correction Theorem (Theorem 6.5.1) of Chapter 5 
shows that every locally checkable protocol that works on a tree topology can be 
stabilized in time proportional to the height of the tree. Thus we can dispense 
with the need for local correctability if the underlying topology is a tree. This is 
achieved by adding actions to do independent checking and resetting of each link 
subsystem in such a way that the correction of a link (u,v) does not invalidate 


the correctness of links above link (u,v) in the tree. 


This theorem also formalizes a useful design technique. First, we construct a 
spanning tree of the network using a stabilizing tree protocol. Next, we design 
a locally checkable protocol to solve the desired problem that works over a tree. 
Finally, we apply the transformation underlying the Tree Correction theorem. 
The stabilizing token passing protocol of Chapter 6 is an example on which this 
design technique could be applied.” 


e Global Correction: The Global Correction Theorem (Theorem 8.1.2) of Chap- 
ter 8 shows that every locally checkable protocol can be stabilized in time pro- 
portional to the number of network nodes. That is we can dispense with the 
need for local correctability or the restriction to a tree topology if we are will- 
ing to incur a stabilization time that is proportional to the number of network 
nodes. This is achieved by adding actions to do independent checking of each 
link subsystem as well as actions to do a coordinated global reset of the entire 


network if a local violation is detected. 


This theorem also formalizes a useful design technique. We construct a locally 
checkable protocol and then apply Global Correction. However, since the sta- 
bilization time of this method is comparatively high, it pays to use the Local 


Correction Theorem whenever a locally checkable protocol can also shown to be 


?However, in Chapter 6 this example is derived using Local Correction. 
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locally correctable. The stabilizing spanning tree protocol of Chapter 6 is an 


example of the Global Correction technique. 


e Stabilizing Compiler for Synchronous Protocols: The Resynchronizer and 
Rollback Compiler ideas in Chapter 9 show that any synchronous protocol 7 with 
time complexity 7, can be converted to an asynchronous, stabilizing version of 
a, with either an additive cost of O(n) in stabilization time or a multiplicative 
factor of T, in storage. Thus for such protocols we can dispense with even the 
need for local checkability. This is achieved by applying global correction to a 
simple synchronizer protocol. While Chapter 9 sketches the main ideas for both 
compilers, the method still requires further development, especially to complete 


and simplify the proofs. 


The synchronous compilers also suggest a useful design technique. Suppose the 
correctness of a task can be specified by an Input/Output relation. Then we 
can construct a synchronous protocol for a given task and then apply one of the 
two compilers. This technique provides stabilizing solutions for many tasks for 
which locally checkable solutions are not known to exists. For instance, this can 
be used to produce efficient and stabilizing minimal spanning tree, min-cost flow 


and coloring protocols. 


Besides the four general methods, we have a useful heuristic. 


Heuristic of Removing Unexpected Packet Transitions: In many cases we 
can make a protocol weakly locally checkable for a good property L (i.e., LZ holds if 
the local properties of all link subsystems hold) by adding a small amount of state. 
To make the protocol locally checkable for E we also have to ensure that each local 
property is closed - i.e., if the local property is ever true, it remains true. This can 
often be done by the method of removing unexpected packet transitions described in 
Chapter 6. The basic idea is that when a node wu receives a packet p from a neighbor v, 
u only accepts the packet if this transition could have occurred in some good state of 
the (u,v) subsystem. If the (u,v) subsystem is in a bad state but the (wu, w) subsystem 
is not, the only way the (u,v) subsystem can affect the (u,w) subsystem is if v sends 
“unexpected” packets to u. Weeding out such unexpected packet transitions is often 


sufficient to ensure that each local predicate remains closed. 


The heuristic of removing enexpected packet transitions is used throughout the 
thesis. It is used in the Token Passing Protocol of Chapter 6, the Reset Protocol of 
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Chapter 7, the Spanning Tree Protocol of Chapter 8, and the Resynchronizer Protocol 
of Chapter 9. 


Comparison with Only Previously Known General Technique 


The only previously known general method for stabilization is the elegant result of 
Katz and Perry [KP90]. How do our methods compare with the general method of 
[KP90]? Recall that in [KP90], checking is done by a single leader node periodically 


taking a snapshot of the entire network. 


e Message Congestion: From a practical standpoint, the most important differ- 
ence is that all our methods have considerably smaller stabilization bandwidth 
than that of [KP90]. Recall from Chapter 9 that stabilization bandwidth is the 
periodic overhead of checking that must be paid even when the protocol is in a 
good state and behaving correctly. Suppose the protocol is to stabilize in time T. 
Roughly, [KP90] has to pay the price of a snapshot of the entire network every 
T units of time. Now a snapshot requires at least O(m) state, where m is the 
number of network links. Thus links leading to the leader node must carry O(m) 
message bits every T units of time. Thus the worst case bandwidth per link is 
very high. By contrast, in our local correction, tree correction, and global cor- 
rection methods, each link only carries a constant number of message bits every 
T units of time. Even the naive synchronous compilers of Chapter 9 have better 
stablization bandwidth than the method of [KP90] because the communication 
overhead of checking is spread out among all the links rather than being concen- 
trated on links leading to the leader. In real networks, each link has a limited 
bandwidth and the worst case bandwidth requirement is an important measure 


of link congestion. 


e Speed: If all links are UDL’s, the method of [KP90] requires O(n) time to stabi- 
lize, where n is the number of network nodes. The local correction method, tree 
correction method, and Rollback compiler can all provide much faster stabiliza- 


tion times. 


e Storage: The method of [KP90] requires O(m) storage at the leader to store 


the snapshot information, where m is the number of links. For fault-tolerance, 
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every node must be prepared to be the leader of the network and be able to 
store O(m) information bits. By contrast, except for the Rollback compiler, the 


storage required by our methods is negligible. 


e Generality: The method of [KP90] is clearly more general than our methods. Our 
methods require protocols to be either locally checkable or to have a correctness 
specification that can be expressed (see Chapter 9) in terms of an I/O relation. 
However, there is at least one case where the method of local checking and 
correction is applicable where the method of [KP90] is not. This is the stabilizing 
end-to-end protocol that we describe in |APV91b]. The problem here is that some 
unknown set of network links may have infinite delay. Thus the global snapshot 
of [KP90] may never terminate. However, it is sufficient ([APV91b]) to do local 


checking and correction on the so-called viable links that have bounded delay. 


To summarize, our methods are less general but more efficient than the method 
of [KP90]. Despite the loss of generality, our methods can be used to stabilize many 


useful tasks, as summarized below. 


10.1.2 Newor Improved Stabilizing Solutions for Specific Prob- 


lems 


Our general techniques provide new or improved solutions for Mutual Exclusion, Net- 
work Resets, Computing Spanning Trees, Topology Update, Minimal Spanning Trees 
and other tasks. We have also applied local correction to an important theoretical 
problem — the problem of end-to-end message delivery in unreliable networks. In this 
problem links can fail continuously — the only guarantee is that there is no cut of 
permanently failed links that separate the sender and receiver. We have not described 


our solution to this problem in this thesis but more details can be found in [APV91b]. 


There are a number of other protocols to which we believe local checking and 
correction can be applied. These include the efficient resource allocation algorithm of 
Awerbuch and Saks ([AS90]), and the elegant virtual circuit protocol due to Spinelli 
([Spi88b]). We hope to produce stabilizing versions of these protocols. We believe our 


general methods provide solutions to many important networking tasks. 
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We emphasize that many of our solutions are practical and can be applied in real 
networks. Messages required for local checking can easily be piggybacked on the “keep- 
alive” traffic sent between neighbors in real networks. Thus solutions based on the 
first three of our general methods (all of which are based on local checking) can be 
added to real networks without appreciable loss of efficiency. Solutions based on the 
synchronous compilers of Chapter 8 can be practical if either the time complexity of the 
underlying synchronous protocol is low (for the Rollback compiler) or if the underlying 
synchronous protocol has an efficient synchronous checker (for the Resynchronizer 
compiler). A disadvantage of the synchronizer methodology is that the method tends 
to slow down the network to the speed of the slowest link. Thus the synchronous 
compiler methodology is best suited for use in networks where all links have roughly 


the same speed. 


Among the most useful practical protocols described in this thesis are the stabilizing 
reset protocol of Chapter 7 (which was briefly tested in a trial implementation on the 
AUTONET) and the spanning tree and topology update protocols of Chapter 8. The 
topology update protocol also illustrates another general paradigm that may be useful 
in practice. We start with a simple protocol P that uses unbounded sequence numbers 
and use global correction to convert P into a stabilizing version P’ that uses bounded 
sequence numbers. In the absence of catastrophic errors, P’ is as efficient (except for 


the small overhead of local checking) and simple as P. 


10.1.3. Modularity Theorem 


The Modularity Theorem (Theorem 3.5.7) is simple but extremely useful. It helps 
us to prove facts about the stabilization of a big system by proving facts about the 
stabilization of each of its parts, as long as each part is suffix-closed. The modularity 
theorem gives us a formal basis for a building-block approach. For example, we can 
start with a stabilizing implementation of a UDL; use this to build a stabilizing reset 
protocol as shown in Chapter 7; and then use the reset protocol as a building block 
to construct a spanning tree protocol (Chapter 8) and a compiler for synchronous 
protocols (Chapter 9). As we have argued at the end of Chapter 4, the requirement 
that each of the parts be suffix-closed is not very restrictive. Essentially, this is because 
our methods tend to produce CIOAs (i.e., automata in which every reachable state is 


a start state) that are suffix-closed. 
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As we build up a complex stabilizing protocol in several layers, the stabilization 
time of the system can be calculated by applying the Modularity Theorem and the 
Transitivity Lemma for behaviors (Lemma 3.2.6). For example, suppose the overall 
system is described by an automaton P. Suppose also that P is identical to the 
stabilizing reset protocol R* of Chapter 7 except that every UDL C,,,, in Rt is replaced 
by a stabilizing implementation of a UDL called C’,,,. Suppose that C/,,, stabilizes to 
the behaviors of C,,, in time t;. Then by applying the Modularity Theorem (which 
is possible because all the constituent automata in Rt are UIOA) we conclude that 
P stabilizes to the behaviors Rt in time t;. But we know from Chapter 7, that RT 
stabilizes to the behaviors of R|L in some constant time, say t2. (Recall that R|L is a 
“correctly working” reset protocol in which all local predicates are true in the initial 
state.) Thus we conclude from the Transitivity Lemma that P stabilizes to R|L in 
time t, + t2. Notice that the stabilization times of each “layer” add up due to the 
Transitivity Lemma. Proceeding similarly, we can calculate the stabilization time of 
a version of the stabilizing tree protocol of Chapter 8 in which the reset protocol R|L 
is replaced by P.? 


Our theorem can be viewed as a generalization of previous modularity results 
[DIM90]. Previous results only applied to the case when a lower layer protocol com- 
puted values of a shared variable that were read by a higher layer protocol. By contrast, 
our theorem applies to more dynamic interaction between the various components of 


a system. 


10.1.4 Modelling 


The models and proof techniques we use in this thesis are based on the large body of 
existing work that has been done in the area of protocol verification. Our model of 
computation is based on the timed I/O automaton [MMT91] which is in turn based 
on the I/O automaton of [LT89]. We have even described the shared memory network 


model in Chapter 4 as an I/O automaton that meets certain restrictions. 


We also use standard proof techniques. We use mapping techniques (Refinement 
Mapping Theorem, Theorem 3.4.3) to show that one automaton has the same behaviors 


as another automaton. We prove that a local predicate is closed using the standard 


3This time, however, we can apply the Modularity Theorem because R|Z is a CIOA and the other 


nodes implementing the spanning tree protocol are UIOA. 
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inductive techniques used to prove program invariants. Perhaps the only unusual 
technique is the Execution Convergence Theorem (Theorem 3.4.5) which is a key tool 
for proving stabilization results. To apply this theorem we have to prove a stability 
condition and a “liveness” condition. We prove the stability condition using standard 
inductive arguments. We prove the “liveness” condition by proving time bounds on the 
occurence of events. We have proved time bounds in a fairly ad hoc, operational way. 
However, there is no reason why the time bounds could not be established by more 
rigorous inductive arguments (as in [LA90]). Thus while the Execution Convergence 
Theorem may appear slightly unusual, it is really a combination of existing verification 


techniques. 


We believe this continuity (with the body of existing work in verification) is an 
advantage of our work. By contrast, previous papers on stabilization have sometimes 
tended to invent new models and proof techniques. Despite our strong linkage with 
existing work, we do have two interesting modelling contributions in this thesis. These 
are the use of a behavior specification for stabilization and the use of unit storage Data 


Links. 


Behavior Specification 


The definitions of stabilization in terms of external behaviors are different from previ- 
ous definitions that are in terms of the states and executions of the underlying automa- 
ton. The external behavior definition allow us to define that automaton A stabilizes to 
another automaton B even though A and B have different state sets. This is most use- 
ful when A is a low level model (e.g., an implementation) and B is a high level model 
(e.g., a specification). This can be done using a definition of stabilization in terms of 
executions if we are prepared to introduce an abstraction function into the definitions 
as in Lamport’s work [Lam83]. However, we prefer the behavior definitions as they 
seem to be more natural; we prefer to use the equivalent of an abstraction function to 


prove behavior stabilization results using the Refinement Mapping Theorem. 


Unit Storage Data Links 


In a stabilizing setting it is necessary to define Data Links that have bounded storage. 


First, as shown in [DIM91a], almost any non-trivial network task is impossible in 
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a stabilizing setting in which the links have unbounded storage and the nodes are 
restricted to be finite state machines. Second, bounded storage models correspond to 


physical reality. 


Thus we use the standard asynchronous message passing model of a computer 
network except that each link is what we call a Unit Storage Data Link (UDL) that 
can store at most one packet. We have chosen unit storage links (UDLs) because 
they are practical (see Section 5.2) and they can be modelled elegantly. We have also 
defined a stabilizing interface to a UDL. This is done by having the link periodically 
deliver a free signal (to avoid deadlock) and by having the sender keep a variable that 
indicates whether the link is free. We hope the UDL model will be used by others. 
The UDL model can easily be generalized to Bounded Storage Data Links by changing 


the free signal to report the number of packets currently stored in the link. 


Earlier papers in stabilization (e.g., [DIM90]) seem to have used shared memory 
models for communication in order to avoid the problems caused by unbounded storage 
links. It does seem very likely (see Open Problems) that protocols that work in the low 
atomicity shared memory model of |[DIM90] can be transformed to work correctly in our 
network model. However, we believe that our protocol descriptions are more accessible 
to designers of “real” network protocols because most “real protocol” specifications 


assume the use of message passing primitives. 


10.1.5 Understanding Existing Work 


The concept of local checking helps in crisply understanding many existing stabilizing 
protocols. Chapter 4 shows that some existing work in the shared memory model can 
be understood crisply in terms of local checking and correction. We believe that many 
existing stabilizing protocols can be understood using three general ideas — one way 


checking and correction, counter flushing, and timer flushing. 


One Way Checking and Correction 


In Chapter 8, we said that a protocol P was one-way checkable if it was locally check- 
able using what we called one-way predicates. Intuitively, a one-way predicate is a 
predicate that involves the state of a node u, the state of any neighbor v of u, and 


the state of the link between u and v. Unlike a general local predicate, a one-way 
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predicate only depends on the state of one of the links between the two nodes. We 
saw that one-way checkable protocols could be checked without the need for a local 
snapshot — it is sufficient for each node to periodically send its state to its neighbors. 
The protocols in [Per83] and [Per85] do checking in this way. 


In some cases, the protocol is also one-way correctable — i.e., we can apply local 
correction to a one-way checkable protocol without the need for a local reset of the link 
subsystem. For example, when v receives a copy of u’s state and detects a violation 
of the one-way predicate from u to v, it may be possible for v to apply a local reset 
function to its own state so as to make the one-way predicate true. Of course, this 
could falsify other “adjacent” one-way predicates. For the protocol to be one-way 
correctable, the dependency relation among the one-way predicates must be acyclic, 
as in the definition of local correctability. For example, the Rollback protocol of 
Chapter 9 is one-way correctable. We speculate that the stabilizing topology update 
protocol of Spinelli-Gallager [SG89] is also one-way correctable 


Counter Flushing 


Suppose a sender wishes to periodically send a REQUEST packet to a set of network 
nodes. The responders must each send back a RESPONSE packet before the sender 
sends its next request. In Chapter 5, for example, we implemented local snapshots 
and resets using such a request-response protocol initiated by the leader of each link 
subsystem. In order to properly match responses to requests, the sender numbers each 
request with a counter. Let m be the number of packets that can be in transit between 
the sender and responder and let n be the number of responders. Then the sender uses 
a counter that has Maz = m+n-+ 1 distinct values. For example, in Chapter 5 we 
used a counter in the range 0...3 because there can be at most two packets in transit 


in a link subsystem and there is only one responder. 


Responders only accept REQUEST packets with a number different from the last 
REQUEST accepted. After accepting a REQUEST the responder sends back a RESPONSE 
with the same number as the REQUEST. The sender retransmits the current REQUEST 
till it receives each matching RESPONSE with the same number. After all matching 


RESPONSE packets arrive, the sender increments its counter. 


The size of Maz ensures that within Maz increments of the counter, the sender 


will reach what we call a “fresh” counter value — i.e. a counter value that is not 
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currently stored in either the links or the responders. We call the method counter 
flushing because the request-response protocol must guarantee the following “flushing” 
property. Suppose the sender sends a request numbered c, where c is a fresh value. Then 
after all matching responses to this request arrive, there must no counter values other 
than c that are stored in the links or at the responders. In other words, the sending of 
a freshly numbered request and the receipt of all matching responses, should “flush” 


the links and responders of “old” counter values. 


In Chapter 5, the flushing condition is guaranteed because the sender and receiver 
are connected by two FIFO links in either direction. Similar forms of counter flushing 
can be used to implement Data Links ([AB89]) and token passing [DIM91a]) in link 
subsystems with bounded storage. Counter flushing is, however, not limited to link 
subsystems. The first example in [Dij74] can be simply understood as counter flushing 
in a unidirectional ring (see Appendix E for more details). Katz and Perry [KP90] 
extend the use of counter flushing to arbitrary networks in an ingenious way. Our sta- 
bilizing end-to-end protocol ([APV91b]) is obtained by first applying local correction 
to the Slide protocol [AGR92] and then applying a variant of counter flushing to the 
Majority protocol of |AGR92]. 


Timer Flushing 


The main idea is to bound the lifetime of “old” state information in the network. This 
is done by using node clocks that run at approximately the same rate and by enforcing 
a maximum packet lifetime over every link. State information that is not periodically 
refreshed is “timed out” by the nodes. In Perlman’s |Per83] topology update protocol, 
timer flushing is used to get rid of erroneous updates that are numbered with the 
maximum possible sequence number. In Perlman’s [Per85] spanning tree protocol, 
timer flushing is used to get rid of “ghost” roots (see Chapter 8 for details.) Spinelli 
[Spi88b] uses timer flushing to build stabilizing Data Link and virtual circuit protocols. 


10.2 Open Questions and Further Problems 


The following is a list of further problems. They are arranged under four categories: 
modelling, increased understanding of local checking and correction, new algorithms, 


and new directions. 
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10.2.1 Modelling 


The following is a list of what we believe are the further problems that are motivated 


by our work in modelling stabilizing protocols. 


e Extend model and theorems of Chapter 3 to allow the use of ran- 
domization and lower bounds on the time between actions: The model 
of Chapter 3 makes no provision for randomization or for lower bounds on the 
time between actions. While the model in Chapter 3 is sufficient for modelling 
asynchronous protocols it cannot be used to model many interesting randomized 
and timing based algorithms. This seems to be an important problem. The hard 
part is extending the model to obtain corresponding versions of the modularity 


and transitivity results. 


e Find a Proof technique for Generalized Stabilization Definitions: The 
definitions of a stabilizing reset protocol and other protocols can be made more 
elegant if we modify the stabilization definitions as follows. We only require that 
the t-suffix of a behavior (execution) be a t-suffix of a behavior (execution) of 
the target set. One problem with this modified definition is that we know of no 
good proof technique to prove that the behaviors (executions) of an automaton 
are suffizes of a specified set of behaviors (executions). While a general proof 
technique for this purpose may be infeasible, it may be possible to find a proof 


technique that works for a large number of cases. 


e Discover stronger variants of the modularity theorem. The modularity 
theorem allows us to infer the stabilization properties of a large system from the 
stabilization properties of its pieces. However, we required that each piece be 
suffix-closed. It is natural to look for weaker or alternate conditions. This may 
be somewhat technical as the suffix-closed requirement does not appear to be 


very restrictive. 


e Find a General Definition of Stabilization Bandwidth: In Chapter 9, 
we intuitively described an important measure for a stabilizing protocol — the 
amount of periodic bandwidth the protocol consumes. The definition we gave 
in Chapter 9 only applies to protocols that have a certain structure. A precise, 


general definition would be very useful. 
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e Find a way of Compiling Stabilizing Protocols in the Shared Mem- 
ory Model into our Network Model. A number of interesting stabilizing 
protocols have been described using the low atomicity, shared memory model 
introduced by [DIM90]. It would be nice to have a compiler that that could 
convert protocols from their model to our network model and vice versa. This 


seems to be feasible. 


10.2.2 Increased Understanding of Local Checking and Cor- 


rection 


Our understanding of local checking and correction is far from complete. This moti- 


vates the following problems: 


e Obtain a better understanding of local checkability. Recall that a protocol 
is locally checkable for some good predicate L, if L is a conjunction of local 
predicates, and each local predicate is closed. Thus we can study this problem 


by asking two separate questions. 


— How much state do we need to add to a protocol so that it’s 
legal states are a conjunction of local predicates? In the token 
passing protocol of Chapter 6 and the reset problem of Chapter 7, we made 
protocols locally checkable by adding a constant amount of state to each 
node. It is natural to ask what the minimum amount of storage is that 


must be added in order to make a protocol locally checkable. 


— Can any weakly locally checkable protocol be transformed into an 
(equivalent) locally checkable protocol? Recall that a weakly locally 
checkable protocol is a protocol whose legal states are a conjunction of 
local predicates; however, the local predicates are not necessarily closed. 
In Chapter 6 we described a simple protocol transformation that consisted 
of removing unexpected packet transitions. In many cases, this heuristic 
is sufficient to ensure that the local predicates are closed. We have used 
this heuristic successfully but we still don’t completely understand when 
this heuristic is guaranteed to work. In Chapter 6, we described a sufficient 
condition called local extensibility. Are there weaker conditions than local 


extensibility? 


271 


e Obtain a better understanding of local correctability. In Chapter 5, we 
used a fairly operational definition of local correctability in terms of a local reset 
function. That definition is adequate for the thesis but it gives little insight into 
the structure of locally correctable protocols. Why are some protocols locally 
correctable but not others? Are there simpler sufficient conditions? It is also 
important to formally understand the connection between local correctability 


and locally checkable protocols that work in dynamic networks. 


e Generalize the Definitions of Locality and Local Checkability in Chap- 
ter 5: In Chapter 5, we defined locality in terms of link subsystems. More gener- 
ally, a subsystem is the composition of the nodes and channels corresponding to 
some subgraph of the network graph. For example if we define locality using the 
subsystems corresponding to the entire network graph, then our method reduces 
to the method of Katz and Perry [KP90]. Another interesting subsystem would 
consist of a single node and all its incoming and outgoing channels. Another 
interesting possibility is to consider subgraphs defined by the sparse network 


partitions [AP90] defined by Awerbuch and Peleg. 


Another simple generalization is to allow more than one local predicate per local 
subsystem. For example, consider an example consisting of two link subsystems 
and four link predicates, L,, £2, 3 and L,. Suppose the system is locally cor- 
rectable using a partial order < such that DL, < Ly < D3 < Ly. Then as we apply 
local checking and correction to this system L, will become true first, followed 
by L» followed by Lz and then L,. Having multiple local predicates per local 
subsystem is only useful if these local predicates are independently ordered in 
the partial order. (If this were not the case we could simply work with a new 
local predicate that is the conjunction of all the predicates.) An example of the 


need for this generality is the Rollback protocol in Chapter 9. 


e Obtain a better understanding of Synchronous Checking In Chapter 9, 
we introduced the notion of synchronous checking, which is a synchronous pro- 
tocol that can check the output of another synchronous protocol. Synchronous 
checking is a little easier than asynchronous checking because all the nodes run 
in lockstep and we do not have to worry about messages in channels. Also, the 
checker need only check the final output of the protocol and not the intermediate 


states. Are there synchronous checkers for minimum spanning tree and max flow 
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protocols that are faster than the protocols being checked? 


10.2.3. New Algorithms 


We suggest the following algorithmic problems: 


e Invent a practical stabilizing reset protocol that stabilizes in O(d) time, 
where d is the actual diameter of the network. By way of comparison, the 
reset protocol of Chapter 7 stabilizes in O(n) where n is the number of nodes 
in the network. This seems to be a major open problem. A solution to this 
problem would yield O(d) stabilizing solutions for the spanning tree, topology 
update, and other problems described in Chapters 8 and 9. To be practical, the 
constants hidden in the O(d) notation must be small (between 1 and 3). 


e Invent a practical stabilizing token passing protocol for rings and ar- 
bitrary graphs: We have already described how to construct a tree from an 
arbitrary graph, and shown how to execute a stabilizing token passing protocol 
on a tree. However, the latency of token traversal on a tree can be quite high, 
for example if the original graph is a ring. The first example in [Dij74] appears 
extensible to token passing in a ring. An efficient stabilizing token passing pro- 
tocol for rings would be useful and practical. Existing token ring protocols such 
as the IKEE 802.3 and FDDI protocols have a number of ad hoc mechanisms to 


deal with failures. 


e Invent a Stabilizing Clock Synchronization Scheme: Fault-tolerant clock 
synchronization schemes typically have to defend against Byzantine faults. It 
would be desirable to invent a practical, stabilizing version of such a protocol. 
This would be an interesting example of a protocol that is robust against both 


Byzantine faults that continue as well as catastrophic faults that stop. 


e Simplify Existing Flow Control Schemes for Transport Protocols and 
Data Links: A flow control scheme is a scheme by which a receiver regulates 
the rate at which a sender sends data in order to prevent buffer overflows at 
the receiver. In [CSV89] we propose an extremely simple stabilizing flow control 
scheme for physical links. It can be considered to be a trivial application of 


local checking and correction to the sender-receiver flow control protocol. The 
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resulting protocol is robust and simple enough to be implemented in hardware. 
Now, robust flow control schemes are a major component of Transport and Data 
link layer protocols. Perhaps existing flow control schemes can be simplified 


using local checking and correction. 


e Invent a stabilizing, distributed protocol to compute Sparse Parti- 
tions: Awerbuch and Peleg [AP90] have shown how to decompose a network 
into what they call sparse partitions. They use sparse partitions to build effi- 
cient solutions for online tracking of mobile users, network synchronization, and 
network routing with low memory. The most efficient distributed algorithm for 
constructing sparse partitions is based on the work of Linial and Saks [LS91]. 
A stabilizing distributed algorithm for sparse partitions would be an extremely 


useful tool. 


e Make the Synchronizer Methodology practical for networks with links 
of different speeds: The synchronizer methodology was introduced in [A we85| 
and is extended to a stabilizing setting in Chapter 9. However, the method 
suffers from a severe drawback in networks in which links have varying delays. 
Essentially, it slows down all links to the speed of the slowest link. It may be 
possible to modify the methodology to avoid this drawback. 


e Invent a stabilizing version of the Bootstrap Protocol for End-to-End 
Communication: Our previous work on stabilizing end-to-end communication 
[APV91b] has concentrated on producing a stabilizing version of the simple and 
elegant Slide protocol [AGR92]. However, the Bootstrap Protocol [AG91] is 
more efficient in some cases, and so a stabilizing version would be of theoretical 
interest. The current solutions are still too inefficient (for example in storage) 


for the end-to-end problem to have any practical application. 


10.2.4 New Directions 


We suggest the following new directions for research in self-stabilization. 


e Local Checking for Randomized Protocols: Local checking of randomized 


protocols is an important research area. There are several randomized protocols 
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(e.g., the Rabin-Lehman dining philosophers protocol of [RL81]) that use ran- 
domization to break symmetry and to guarantee termination. Such protocols 
can often be locally checked in a deterministic fashion. For example, in graph 
coloring, it is easy to check whether a neighboring node has the same color. If, 
however, the check reveals a problem, a randomized local reset action must often 
be applied. Thus in graph coloring, a node may randomly choose a new color 
once it discovers that it has the same color as one of its neighbors. Now the 
randomized local reset functions can be applied at a node after all other nodes 
have already chosen their colors. Thus the dynamics (and the analysis) of the 
stabilizing protocol may be quite different from that of the original randomized 


protocol. Preliminary work in this area has been done by Baruch Awerbuch, 


Leonore Cowen, and Mark Smith at MIT. 


Protocols whose behavior is correct only with high probability are even harder 
to check locally. Suppose that a (non-stabilizing) randomized protocol computes 
sparse partitions such that with high probability the partitions have low diam- 
eter. Then even if local checking can detect “poor” partitions, how can we tell 
whether the partitions are bad a) because of errors in the initial random bits at 


the nodes or b) because of a low probability outcome of the protocol? 


Stabilizing Data Structures: We have suggested earlier that programs that 
run on a single shared memory can be made stabilizing by using domain re- 
striction — the states of the program are restricted to legal states. Consider the 
problem of building a stabilizing data structure (e.g., a dictionary or a queue) 
within the memory of a single processor. The data structure provides a certain 
set of operations (for example to insert and delete elements). The correctness 
of the data structure can be defined in terms of allowable sequences of opera- 
tions. This notion of correctness is similar to the behavior specifications of I/O 
automata. Thus we can define a stabilizing data structure to be one in which 
the data structure can begin in an arbitrary state; however, any sequence of op- 
erations on the structure will eventually have a suffix that is a correct sequence 
(or a suffix of a correct sequence) of the data structure. For example, in a stabi- 
lizing dictionary, we would require that any elements inserted after the structure 


stabilizes would be found when searched for. 


Now any such data structure can be trivially made stabilizing in the following 


way. Before any operation on the data structure, we first check the invariants of 
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the data structure, and reinitialize the data structure if an error is found. Un- 
fortunately, this slows down regular operations. Thus if we charge for processing 
(which we have not done in the distributed model in this thesis), there appears 
to be a tradeoff between stabilization time and the time to complete normal 
operations. For some data structures, the tradeoff can be extremely good. For 
example, we can easily implement a stabilizing, bounded size queue using a fixed 
size array. The head and tail pointers can be restricted to stay within the array 
bounds. However, it appears to be much harder to obtain good tradeoffs for say 
a tree-based implementation of a dictionary. We have done some preliminary 


thinking in this area [MPV90]. 


e Applying Self-Stabilization to Other Areas: Most of the stabilizing pro- 
tocols described in this thesis are used for routing, scheduling, and resource 
allocation — tasks typically found in the network and Data Link layers of the 
communication hierarchy. However, in principle there is no reason why self- 
stabilization cannot be added as an “extra envelope” of fault-tolerance for higher 
layer protocols — e.g., file transfer and database protocols. Such protocols should 
be designed to avoid errors in the case of common faults such as node and link 
crashes. However, it is also desirable that these protocols recover by themselves 
after catastrophic errors. Self-stabilization is also applicable at the lowest layer 
of the protocol hierarchy. For example, it would be desirable to have stabilizing 


clock recovery and framing protocols at the physical layer. 


e Generalized Local Checking: The method of Katz and Perry [KP90] consists 
of a single leader that checks the entire network. In our method of local checking, 
nodes independently (and in parallel) check each link subsystem. The Katz and 
Perry method is more general but less efficient. Local checking of link subsystems 
is efficient but only applicable to locally checkable protocols. Thus there may 
be intermediate approaches in which the notion of locality is generalized to an 


arbitrary set of subnetworks of the original network. 


10.8 Summing Up 


Self-stabilization abstracts the ability of a protocol to tolerate catastrophic faults 
that stop. On the other hand, the cost of self-stabilization is often low. Thus self- 
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stabilization can provide a cheap way to improve the fault-tolerance of network pro- 


tocols. 


Local checking can be used to design efficient, stabilizing protocols. The resulting 
protocols can be proved correct in a systematic way. The overhead of local checking can 
be piggybacked on existing keep-alive traffic between network nodes. Local checking 
may prove to be a useful debugging tool because it provides a continuous check of 


system predicates, and violations can be logged. 


Our research into stabilization and local checking has helped us better understand 
existing protocols, both stabilizing and non-stabilizing. For instance, in the process of 
understanding Global Correction, we realized that a Global Reset protocol provides 
a mating relation that is a generalization of the guarantees of Data Link protocols. 
We also saw that many existing protocols use a special form of local checking that we 


called one-way checking. 


Whitehead once said that the interest of a generalization is the interest of a road for 
those who know what travel is; and the pleasure of the road has its roots in the labor 
of the journey. After struggling to understand many examples of stabilizing protocols, 
we have begun to appreciate the simplicity of understanding provided by such concepts 
as local checking and counter flushing. They have helped us to see things a little more 


clearly and to travel a little distance. We hope that our readers will go much further. 
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Appendix A 


Notation 


We summarize notation that is commonly used in this thesis. Other definitions can 
be found using the index. For example, if the notation for C,,, refers to a UDL, the 
definition of a UDL can be found by using the index. 


e {u,v}: For an pair of neighboring nodes uw and v, this denotes the unordered 
pair corresponding to the directed edge (u,v). This is useful in defining a partial 


order on unordered pairs of nodes. 


e A|£Z: For any automaton A and any subset L of the states of A, A|L is the 
automaton that is identical to A except that the set of start states of A|L is L. 


e Cy: The Unit Capacity Data Link (UDL) corresponding to directed edge (u, v) 
in a topology graph. 


e Conj(L): For any link predicate set £L, Conj(£) denotes the conjunction of the 


predicates in L. 


e A(t): For any network automaton NV, V(t) is the automaton that is identical to 
N except that the link and node delays in M(t) are equal to ft. 


e Quv: The packet stored on the UDL corresponding to directed edge (u,v) in a 
topology graph. If the value of the variable is nzl, then there is no packet stored. 


e queue,|v|: A queue of packets for outgoing edge (u,v) in the node automaton 


corresponding to node wu. See the definition of a node automaton. 
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e zqueue,|v|: The concatenation of Qu, and queue,|v|. Only used in some proofs. 


Can be the read as the extended queue at node wu corresponding to neighbor v, 


e U(A): For any automaton A, U(A) is the automaton that is identical to A except 
that the set of start states of U(A) is equal to the set of states of A. Thus U(A) 
creates a UIOA from an IOA. 


The following symbols are used frequently to denote the following quantities 


e <: A partial order on a set of local predicates. 
e a: An execution of an automaton 

e a;: The j-th action in a behavior or execution. 
e @: A behavior of an automaton. 

e f: A reset function. 

e G: A graph or a topology graph. 

e L: A predicate (set of states) of an automaton. 
e Lu. A local predicate for edge (u,v) 

e L£: A set of local predicates. 

e A’: A network automaton. 

e Nt: An augmented network automaton. 

e P: A packet alphabet. 


e s: A state of an automaton. A state with a subscript like s; often denotes the 


i-th state in an execution. 


e ¢: A time. Times with subscripts like t,, t, denote special constants; many of 


these are listed in the index. 
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Appendix B 


Proofs for Chapter 5 


Let N*+ = Augment(N,L,f). We will use s and 5 to denote states of V*. In deriving 
time bounds recall that all locally controlled actions at nodes take ¢,, time; also for 


any link all actions have upper bound ¢,. 


B.1 Any Execution of \’~ is infinite 


Our first lemma states that all executions of Nt are infinite and hence in all execu- 
tions of N+, time grows without bound. This follows because we have ruled out the 
possibility of so-called Zeno executions in our model (see Chapter 2). We will assume 


this implicitly in what follows without making explicit reference to this lemma. 
Lemma B.1.1 Any ezecution a of N+ is infinite. 


Proof: Suppose not. Then there is some final state s of a after which no action takes 
place. Consider any channel C,,,. If s.Q., = nil, then a FREE, action is enabled; 
but s.Quy» = p # nil, then a RECEIVE,,,(p) action is enabled. Both cases contradict 


the assumption that s isa final state. J 


Notice also that since V+ is a UIOA, any suffix of an execution of V+ that begins 
with a timed state is also an execution of NV. We will assume this implicitly when we 


apply some of the lemmas and claims described below. 
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B.2 Basic Properties of links 


The first lemma states that once a link is drop-free (see Definition 5.6.7), it remains 


drop-free. 


Lemma B.2.1 For any (u,v) and any transition (s,7,8) of Nt, if s © Fu», then 
8 E Fyy. 


Proof: If s.freeq,|v] = true then since s € Fu», s.Quy = nil. Thus the only action 
that can cause 5.Q,, 4 nil is a SEND,,(p) event that also sets &.freeq,|v] = false. 
If s.freeq,,|v| = false then the the only action that can cause 3.freeq,|v] = true is a 


FREE,,, event which in turn can only occur if s.Qu, = 85.Quy = nil. J 


The next lemma says that (u,v) becomes drop-free after the first action that sends 


a packet to C,,, or the first signal from C,,, that it is free. 


Lemma B.2.2 For any edge (u,v) and any execution a, (u,v) is drop-free in all states 
of a that follow a FREE,, or a@ SEND,y,»(p) action. 


Proof: After a FREE, action, Qu, = nil, and after a SEND,,,(p) action, freeg, |v] = 


false. Hence if s follows either of these actions in a, then s; € F,,. The lemma follows 
from the stability of F,,, (Lemma B.2.1). I 


Once a link is drop-free we can be sure that all packets sent on the link will be 


delivered. 


B.3 Time Bounds for Correction Phases 


Recall the definition of a correction phase on a link from 5. This section builds up to 


a few important time bounds that are useful in the main body of the proof. 


Only the last two lemmas are important. The remaining claims are only useful 
in proving the last two lemmas. In the following claims, when we say that an event 
occurs within time ¢ in execution a, we mean that the event occurs within time t of 


the first state in a. 


The first claim says that a packet on a link will be delivered in ¢; time. 


281 


Claim B.3.1 For all executions a and any (u,v), if in the initial state, Quy = p # nil, 
then a RECEIVEu,»(p) event occurs within t; time units. 


The next claim says that a sender will know that a link is free in 2t; time. Also 
after this period, the link will be drop-free. (Intuitively it takes t; time units to deliver 
any packet on the link, and ¢; time units to deliver the FREE signal to the sender.) 


Claim B.3.2 For all executions a and any (u,v), within 2t; time units there is a state 


s such that s.freeq,|v] = true and (u,v) is drop-free in s. 


Proof: By Claim B.3.1 in ¢; time units after the first state of a, there must be a 
state s, such that s.Qu, = nil. Now we have two cases. First, suppose within t, 
time units after s a SEND,,(p) event occurs. Then in the state preceding this event, 
freeq,,|v] = true and Qu, = nil and we are done. On the other hand if the first case 
does not occur, then Q,, = nil for t; time units after s. Hence in ¢; time units after 
sa FREE, action must occur, resulting in a state § such that &.freeq, |v] = true. By 


Claim B.2.2, (u,v) is drop-freein 5. ff 


The next claim bounds the time before a response will be sent. Note that in the 
code, responses are sent continuously even without waiting for a request. This helps 
avoid deadlock in arbitrary initial states. We rely on the matching process to weed 


out meaningless responses. 


Claim B.3.3 Response Time: For all executions a and any leader edge (u,v), there 


must be some SENDy,u(Presp) action in a that occurs in 3t, + 4t; time units. 


Proof: From Claim B.3.2, within 2¢; time, there is a state s such that s.freeq,|u] = true. 
If queue,|u] is empty for t, time units after s, then a SEND, u(Presp) will occur in this 
period. If not, gueue,|w] becomes non-empty within time t, after s, which enables the 
sending of a data packet. In this case, within 2t,, time units after s, a SENDy,u(Ddata) will 
occur, resulting in a state 5 in which turn,|u] = response. From Claim B.3.2, in 2 ¢; time 
units after 5, there is a state s’ such that s’.freeq,|u] = true and turn,|u] = response. 


Finally, in ¢,, time units after s', a SENDyu(Presp) will occur. 


The next claim bounds the time before a new phase will start on a link. 
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Node u Node v 


Time for link 
to become drop-free 


Request 
3t, + 4t, 
(From Response Time Claim) 


t 1 
Response 


Figure B.1: Summary of the proof of Claim B.3.5. 


Claim B.3.4 For all executions a and any leader edge (u,v), within 2t, + 2t; time 


units there is a state in which phase,,|v| = true. 


Proof: Assume phase,,|v| = false in the initial state of a or we are done trivially. 
From Claim B.3.2, within 2¢; time, there is a state s such that s.freeq,|v| = true. If 
queue, |v] is empty for t, time units after s, then a SENDz»(Preq) action occurs within 
this period, resulting in a state in which phase,,|v] = true. On the other hand, suppose 
that queue,,|v] becomes non-empty within time t, after s. Then a SEND.,»(Pdata) event 
will occur within 2¢, time units after s, resulting in a state in which phase, |v] = true. 


The next claim bounds how long it can take for a phase on a link to complete once 
it has started. 


Claim B.3.5 For all a and any leader edge (u,v), within 4t, + 10t; time units there 


is a state in which phase,|v| = false. 


Proof: The proof is summarized in Figure B.1. We assume the claim is false, so 
phase,,|v| = true for 4t,, + 10t; time units after the start of a. First, from Claim B.3.2 


in 2t; time, there is a state s such that both (u,v) and (v,u) are drop-free in s. Thus 
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any any requests and responses that are sent in states after s will be delivered to the 


corresponding receiver. 


From Claim B.3.2, within 2¢; time after s, there is a state s’ such that s’.freeq, |v] = 
true. Thus a a SENDu»(Preq) action is enabled in s’. Thus within ¢, time units after 
s’ a SENDu»(Preq) action occurs. In t; time units after this SENDuz»(Preq) action, a 
RECEIVE,» (Preq) action occurs resulting in a state say s”. Also in all states after s”, 
count,|u] = count,|v]. Then by Claim B.3.3 in 3t, + 4t; after s” a SENDyu(Dresp) 
event occurs with presp.count = count,|v]. Finally, another ¢; time units after this 
event, a RECEIVE, u(Presp) event occurs with presp.count = count,|v|. In the state 
immediately after this event, phase,|v] = false. This contradicts the assumption that 
phase,,|v| = false for 4t,, + 10t; time units after the start ofa. I 


All the previous claims are only useful in proving Lemma 5.6.5 and Lemma 5.6.6. 


We now prove Lemma 5.6.6. 


Proof: First assume that (u,v) = u. Now we consider two cases. If so.phase,,[v] = 
false then by Lemma B.3.4, within 2t,, + 2¢; time after so, there is a state s; in which 
s;.check,|[v] = true. However, the action before s; must be a SENDu»(Paata) event. (The 
only other action that sets check,|v| = true is a SENDy»(Preq) action, but such an action 
is not enabled if queue,|v| is non-empty and phase,,|v| = true.) If so.phase, |v] = true, 
then by Lemma B.3.5 within 4¢, + 10¢; after s9 we reach a state in which phase, |v] = 
false and we are back to the previous case. Thus in both cases, a SENDu,»(Pdata) Occurs 


within t, after so. 


Now suppose that [(u,v) = v. Once again we have two cases. Suppose s9.turn,[v] = 
data. From Claim B.3.2, within 2¢, time after so, there is a state s; such that 
s;.freeq,|u] = true and s;.turn,|v| = data. Thus within t, time after s; a SEND.» (Paata) 
event occurs. If so.turn,|v] # data, then by Lemma B.3.3, we see that in at most 
3t, + 4t; after s;, a SENDyu(Presp) event occurs that will result in state in which 
turn,[v| = data. Then we are back to the first case. In both cases, a SENDu,»(Pdata) 


occurs within ¢, after so. J 


B.4 Proof that Clean Edges remain Clean 


The following claim is useful in determining how and when the counter values stored 


in the links and the receiver can change. Recall that we used undefined when there is 
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no counter value stored on a link. 


Claim B.4.1 In any execution, and for any leader edge (u,v), the only action that 


can add a new element that is not undefined to countset(u,v) is a SENDu»(Preq) action. 


Proof: The only other actions that affect countset(u,v) are a RECEIVE »(Preq) ; 


SENDy,u(Presp) and RECEIVEy,u(Presp) , none of which can add any new values other 
than undefined. Jf 


A nice property is that once an edge becomes clean, it remains clean. This is stated 
in Claim 5.6.11. We now prove Claim 5.6.11. 


Proof: We assume that leader edge (u,v) is clean in s and we show that it remains 
clean in 8. We will refer to the five predicates in the definition of a clean link by their 


numbers. The stability of the first predicate follows directly from Lemma B.2.1. 


If s.phase, |v] = false and §.check,|v| = false, then no SEND,» (Preq) action can occur 
which by Lemma B.4.1 is the only action that can change countset(u,v). Thus the 
second predicate holds and the remaining three hold trivially. If s.phase,,|v] = false and 
5.check,|v] = true, then 7 must be a SENDy»(Preq) OF a SENDu,»(Pdata) event which will 
cause Predicate 3 to hold. Also by Predicate 2 in state s, s.count,|v] # s.respcount(u, v) 
and s.count,|v| # s.count,[u]. Since a SENDu»(Preq) OF @ SENDu,»(Pdata) event does not 
change count,|v] or respcount(u,v) or count,|u], Predicates 4 and 5 hold in §; also 
Predicate 2 holds trivially. 


If s.phase,,|v] = true and 8.check,|v] = false, then 7 must be a RECEIVEy,u(Presp) 
action and s.count,|v| = respcount(u,v) and 8.count,|[v]| = (s.count,|v] + 1) mod 4. 
Then we can use the fact that Predicates 3 and 4 hold in s to infer that Predicate 2 
holds in 5. Also Predicates 3, 4, and 5 hold trivially in 3. If s.phase,|v] = true and 
5.check,|v| = true, then the only actions of interest are SENDu.»(Preq) (which makes 
Predicate 3 true and leaves the others true), SENDy,u(Presp) (which makes Predicate 4 
true and leaves the others true) and a RECEIVE,» (Preq) - For a RECEIVEu»(Preq) » WE 
can use the fact that Predicate 3 holds in s to infer that Predicate 4 holds in s. All 
other Predicates hold trivially. Note that 7 cannot be a SENDz»(Ddata) event because 


s.phase,|v]| = true. ff 
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B.5 How Links Become Quiet: Detailed Proofs 


Recall the definition of a quietlink from Chapter 5. For (u,v) to be quiet, we want 
to show not only that L,,, holds by the end of the second (u,v) phase but also that 
no more “reset” actions can occur after this point so that LD, will remain true. This 


motivates the following definitions of a reset transition. 


Definition B.5.1 Consider any transition (s,7,5) of Nt. We say that this transition 


is a reset transition at node u with respect to node x if: 


e s.mode,|z] = reset AND 


e Either m is @ SENDus(Presp) or 7 18 @ RECEIVEs,u(Presp) and I(u,x) = x. 


Intuitively, a reset transition at node u causes the local reset function f to be 


applied to the state of wu. 


We start by proving 5.6.17, which is the stability condition for a quiet link. Proof: We 
will show that each of the five predicates used to define a quiet link (Definition 5.6.15) 
holds in &. We will refer to the four predicates using the numbers given in Definition 


5.6.15. The first predicate holds in § because of Claim 5.6.11. 


Consider the second predicate. We wish to show that (8|u, 3|(u,v), 8|(v,u), |v) € 
Ly. We know that the second predicate holds in s. We also know that if 7 is any 
action of NV’, then the second predicate holds in § as well because L,,, is a closed 
predicate by definition. The only other transitions of V* that can affect L,, are reset 


transitions at u or v. 


Suppose (s,7,8) is a reset transition at u. From the fact that the third predicate 
holds in s, it is easy to see that (s,7,5) cannot be a reset transition at node u with 
respect to v. Suppose (s,7,8) is a reset transition at node u with respect to some 
neighbor « # v. Let e denote the leader edge corresponding to edge (u,z). Then e 
is not quiet in state s. From the hypothesis, we infer that {u,z} < {u,v}. Thus by 
the stability condition for local correctability, and since s|u = f(s|u,2), we infer that 
the second predicate holds in & even if (s,7,8) is a reset transition at u. A similar 


argument works if (s,7,8) is a reset transition at v. 


286 


We can infer that the third predicate holds in § from the fact that the third, fourth 
and fifth predicates hold in s. The only actions to consider are RECEIVEy,u(Presp) and a 
RECEIVE,» (Preq) - We can infer that the fourth predicate holds in § from the fact that 
the third and fourth predicates hold in s. The only action to consider is a SENDu»(Preq) 


action. 


Consider the fifth predicate. Let X be the predicate Quy» = Presp aNd Presp.count = 
count,|v]. The only actions that can change the truth or falsity of predicate X are 
SENDy,u(Presp) OF RECEIVE, u(Presp) - Suppose that either 7 is a SEND, u(Presp) action 
and s.count,|v| # s.count,|u] OR a is a RECEIVEyu(Presp) action. Then X will become 
false in § and the fifth predicate holds trivially. On the other hand, suppose 7 is 
a SENDyu(Presp) action and s.count,|v| = s.count,[u] Thus X will become true in 3. 
However, in this case, from the definition of a clean link, s.Q.u. ¢ Paata. Thus 7 will 
make the fifth predicate hold in s. 


Thus, suppose that X is true in s and also in 8. The only other actions to consider 
are actions that change the state of s|u. We know that if 7 is any action of NV, then the 
fifth predicate holds in § as well because L,,,, is a closed predicate by definition. The 
only other transition of V* that can affect the fifth predicate is a reset transition at wu. 
Suppose (s,7,8) is a reset transition at node u with respect to some neighbor z F v. 
Let e denote the leader edge corresponding to edge (u,a). Then e is not quiet in state 
s. From the hypothesis, we infer that {u,x} ¢ {u,v}. Thus by the stability condition 
for local correctability, and since 5|u = f(s|u, x), we infer that the fifth predicate holds 


in § even if (s,7,8) is a reset transition at u. JJ 


Next we show the required liveness condition: that a leader edge (u,v) will become 


quiet in bounded time if all leader edges less than (u,v) are already quiet. 


In the rest of this section, we prove Lemma 5.6.18, which describes how and when 
links become quiet. A quick overview of this section can be obtained by skimming 
Claim B.5.4, Figure B.2, Definition B.5.7 and the last three lemmas in the section. 


We start by proving some more detailed properties of the local snapshot and reset 
protocols during a clean phase. For all the claims and lemmas in this section, we fix 
an execution a of V’* and a leader edge (u,v). When we refer to a (u,v) phase we 
mean a (u,v) phase in a. When we refer to the code, we mean the code in Figures 5.6 


and 5.7. 
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Node u Node v 


START OF PHASE . TIME 
——e! b 
aN 
Midpoint m 
a 
Cc 


Penultimate State s’ ee 
END OF PHASE 


Figure B.2: More detailed structure of a clean phase 


Since neither the mode or the counter at the leader can change except at the end 


of a phase it makes sense to talk of the mode and counter value of a phase. 


Definition B.5.2 Consider any clean (u,v) phase P with first state r. Then we use 


mode(P) to denote r.mode,|v| and count(P) to denote r.count,|v]. 


Claim B.5.3 For all states s in a clean (u,v) phase except the last state, s.count,|v] = 


count(P) and s.mode,,|v] = mode(P). 


Proof: From the code, the values of count,|[v| and mode,,|v] can only change at the 
end of the phase. Jf 


Figure B.2 shows the structure of a clean phase in more detail for some leader 
edge (u,v). The next claim formalizes the intuition behind Figure B.2 and defines two 


important states within a phase: the midpoint and the penultimate state. 


Claim B.5.4 Any clean (u,v) phase P contains the following four actions in the order 


shown: 
e A SENDu»(Preqg) action with preq.count = count(P) and preq.mode = mode(P). 
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e A RECEIVEu,»(Preq) action with preq.count = count(P) and preq-mode = mode(P). 


e Exactly one SENDynu(Presp) action with Dresp.count = count(P) and 
Presp-node_state = m|v, where m is the state immediately following this action. 


We will call m the midpointphase!midpoint of the phase. 


e Exactly one RECEIVE, «(Presp) action with Presp.count = count(P) and Presp node_state = 
m|v, where m is the midpoint of the phase. We will call the state immediately 


before this action, the penultimate statephase!penultimate state of the phase. 


Proof: The four actions are shown in Figure B.2 as a, b, c, and d respectively. The 


midpoint and penultimate states are also marked. 


Informally, the claim follows from two facts. First since both (u,v) and (v,u) are 
drop-free in all states of P any SEND,,,(p) action in P must be followed by a state 
in which Qu» = p. (i.e., any packet sent on channel C’,,, will be stored in the link.). 
A similar statement holds for link (v,u). The second fact is that, by definition, at 
the start of a clean phase, count,|v| ¢ countset(u,v). Thus at the start of the phase, 
there are no potentially confusing requests or responses numbered with the value of 
the counter at the sender. This ensures, for instance, that when a response Presp 18 
received at u with presp.count = count,|v|, then presp was sent in the phase. Similarly 
it ensures that when a request preg is received with p,¢g.count = count,|v], then the 


request was sent in the phase. 


The formal argument is quite tedious. We start by proving the last statement in 
the claim and then working backwards to prove the other three. Let’s sketch the first 


part of the argument. 


We know that P can only end with a RECEIVE, u(Presp) With Presp.count = count,|v]. 
Thus in the state s’ before this action, respcount(u,v) = count,[v]. But in the first 
state of P, we know that since P is clean, respcount(u,v) # count,|[v|. Thus there must 
have been a SENDy,u(Presp) action in P. Also, there can be only one such action in P: 
any earlier SEND,yu(Presp) action would have caused the phase to have ended earlier; 
no later SENDy,u(Presp) action can occur because Q,,, # nil for all remaining states in 
the phase except the last state. If the state after the only SEND, u(presp) action is m, 
then from the code pyesp.node_state = m|v. Similar arguments can be used to show the 


remainder of the claim. J 
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In Figure B.2, in all states following 6 and before action c, mode,|u] = mode,,|v}. 
Informally, this is because mode,[u] is set to the mode carried in the request packet and 
cannot change until action c occurs at which point mode,[u] is changed to snapshot. 


Then mode,|u] remains at this value till the end of the phase. Formally: 


Claim B.5.5 Consider a clean (u,v) phase P with midpoint m. Then in the state 


before m, mode,|u| = mode,|v| and in all remaining states in P, mode,|u] = snapshot. 


Proof: After the first RECEIVE.,»(Dreq) action in P, mode,|u] becomes equal to mode,|v]. 
The value of mode,|u] can only change due to RECEIVE,,»(Preq) actions and SENDy,u(Presp) 
actions. However, future RECEIVE,» (Preq) actions in P cannot change mode,|u] be- 
cause from the definition of a clean link, p,<g.count = count,|u]. After the first (and 
only, see Claim B.5.4) RECEIVE,,u(Presp) action, mode,|u] = snapshot and remains at 
that value till the end of the phase. [J 


The following definition is convenient: 


Definition B.5.6 The leader edge corresponding to an edge (u,v) is (u,v) if I(u,v) = 
u and (v,u) if I(v,u) =v. 


In order to guarantee correction or checking at end of a (u,v) phase, we need 
restrictions on the states of links adjacent to either u or v. Otherwise, concurrent 
checking/correction on these adjacent links may invalidate the checking/correction 


done in the (u,v) phase. 


Definition B.5.7 We will call a (u,v) phase P well-behaved if P is clean and for 
all states s in the phase and for all {w,z} < {u,v}, the leader edge corresponding to 


{w,x} is quiet in s. 


Now in a well-behaved phase, the response contains the state of v at the midpoint. 
However, the response is received in the penultimate state. Despite the fact that the 
state of v recorded in the response is “old”, the recorded state is stil useful in the 


following precise sense. 


Claim B.5.8 Phase Invariant: Consider any clean (u,v) phase P with midpoint 
m and penultimate state s'. Then for all states r in the interval |m, s'|, the following 


predicates hold in r: 
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© 7.Quu = Presp ON Presp.count = count(P). 


e For any y: If (y, nil, nil, ppesp.node_state) € Ly, then (y, nil, nil,r|v) € Lu» 


Proof: The first part of the claim follows directly from Claim B.5.4. We establish the 
second predicate by induction on the length of the interval [m, s’]. First, this predicate 


is true in m because M.Qy,u = Presp ANd Presp node_state = m|v by Claim B.5.4. 


Next, assume the second predicate holds in some state r’ before r and consider the 
transition (r’,7z,r). The only actions of interest are the actions that change the state 
of v. If wis any action of NV, then the second predicate holds in r as well because L,,», is 
a closed predicate. The only other transitions of V* that can affect the state of v are 
reset transitions at v. We first see that (r’,7,r) cannot be a reset transition at node 
v with respect to u. Suppose (r’,7,r) is a reset transition at node v with respect to 
some neighbor z # u. Let e denote the leader edge corresponding to {v,z}. Then e is 
not quiet in state r’. But since the phase is well-behaved, we infer that {v,z} ¢ {u,v}. 
Thus by the stability condition for local correctability, and since rlv = f(r’|v,x), we 


infer that the second predicate holdsinr. J 


We can now state two lemmas that explain why the local snapshot and reset pro- 
cedures work correctly. Let us call a snapshot phase a well-behaved phase P such that 
mode(P) = snapshot. Similarly a reset phase is a well-behaved phase P such that 
mode(P) = reset. The first lemma states that if L,, does not hold at the end of a 
snapshot (u,v) phase, then at the end of the phase mode,|v|] = reset. This ensures 


that the next (u,v) phase will be a reset phase. 


Lemma B.5.9 For any snapshot (u,v) phase P with last state s the following is true. 
If (s|u, s|(u,v), s|(v, wv), s|v) ¢ Lu», then s.mode,|v] = reset. 


Proof: Let s’ be the penultimate state immediately before s in P. The action just 
before s is a RECEIVEy,u(Presp) action. Since s’ is the penultimate state of a snapshot 


phase, mode, |v] = snapshot and Qyu = Presp ANd Presp-count = count,|v| in s’. 


Thus, from the code, s|u = s'|u and s|v = s’|v (i.e., the basic states of nodes u and v 
remain unchanged after the RECEIVE,,u(Presp) action.). Next, from the fact that (u,v) 
is clean in s’, we deduce that s|(u,v) = s’|(u,v) = nil. Also, after a RECEIVE, u(Dresp) 


action s|(v,u) = nil. 
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Thus if (s|u, s|(u,v), s|(v,u),s|v) ¢ Lu» then (s'|u, nil, nil, s'|v) ¢ Lu». But by the 
phase invariant (Claim B.5.8) if (s’|u, nil, nil, s'|v) ¢ Ly, then we can be sure that 
(s'|u, nil, nil, presp-node_state) ¢ Lyy. Thus if (s|u,s|(u,v),s|(v,u),s|v) ¢ Lu,» then 
(s'|u, nil, nil, Dresp-node_state) ¢ Ly. Hence, from the code of the RECEIVE, «(Presp) 


action it is easy to see that s.mode,|[v] = reset. JJ 


The second lemma states that at the end of a (u,v) reset phase, (u,v) is quiet. 
Lemma B.5.10 For any reset (u,v) phase P with last state s, (u,v) is quiet in s. 


Proof: Let s’ be the state immediately before s in P. The action just before s must 
be a RECEIVE, u(Presp) action. Since s’ is the penultimate state of a reset phase, 
mode,|[v]| = reset and Qvyu = Presp and Presp-count = count,|v| in s’. Let m be the 
midpoint of P and m’ be the state immediately before m in P. By Claim B.5.5, 


m'.mode,|u] = m'.mode,|v] = reset, and hence presyp.node_state = f(m'|v,u). 


Now from the correction property of a local reset function, (f(s'|u,v), nil, nil, f(m'|v,u) € 
Lu». Since presp-node_state = f(m'|v,u), (f(s'|u,v), nil, nil, presp-node_state) © Lyy. 
But by the phase invariant, Claim B.5.8, this implies that (f(s'|u,v), nil, nil, s'|v) € 
Ly. But by similar arguments as in the proof of the previous lemma: s|(u,v) = nil, 
s|(v,u) = nil, and s|v = s’\v. However, since s'.mode,|v] = reset, s|ju = f(s'|u). 


Together, these equations imply that (s|u, s|(u,v), s|(v, wv), s|v) € Lu». 
Next, s.mode,|[v] = s.mode,|u] = snapshot by Claim B.5.3 and Claim B.5.5. Also, 


since (u,v) is clean in state s’, if 5.Quv = Preq, then Preg-count = s.count,(u]. Now 
it is easy to verify now that all five predicates used in the definition of a quiet link 


(Definition 5.6.15) hold in states. [J 


We are now ready to prove Lemma 5.6.18. The proof is almost immediate from 
the last two lemmas. We recall the statement of Lemma 5.6.18. If every leader edge 
(w,x) < (u,v) is quiet in some state s; of some execution a of NT|C, then (u,v) is 
quiet in some state that occurs within 3-¢, of s; in a. 

Proof: First from the hypothesis and Lemma 5.6.17, every (u,v) phase in a is well- 
behaved. Consider the first (u,v) phase P in a that starts after state s;. If mode(P) = 
reset, then we know from Lemma B.5.10 that at the end of P, (u,v) is quiet. If 
mode(P) = snapshot, then we know from Lemma B.5.9 that at the end of P, mode,|v] = 


reset. Consider the next (u,v) phase in a, say P’. Since mode,|v] only changes at the 
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end of a phase, mode(P’) = reset. Thus by Lemma B.5.10, (u,v) is quiet at the end 
of P’. 


Hence there is some state s; that occurs before the end of the second (u,v) phase 
that follows s; in a and such that (u,v) is quiet in s;. Thus by the phase rate lemma, 


Lemma 5.6.5, s; must occur within 3¢, time units after s; ina. Jf 
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Appendix C 


The AAG reset protocol 


In this chapter, we describe why three phases were used in the original AAG protocol 
[AAG87] and also describe the changes that were required to convert the AAG protocol 
into the reset protocol described in Chapter 7. We also show that the mating relation 
provided by the AAG protocol is not transitive. 


C.1 Why three phases are used in the AAG proto- 


col 


The AAG reset protocol [AAG87] is much more conservative than the Simple Reset 


Protocol about allowing a node to to return to Ready mode. 


The point of all the conservatism in the AAG protocol is as follows. The use of 
three phases ensures that if reset requests stop being made, all nodes will eventually 
return to Ready mode. The AAG protocol makes this guarantee even in dynamic 
networks. The additional rules the AAG protocol uses to work in dynamic networks 
are remarkably simple. Suppose a link from node wu to node v fails. If node v is node 
u’s parent (in the abort tree), node u takes over as the root of the abort tree; if node 
u is expecting an ack from node v, node wu assumes it has got an ack from v. Finally, 


when a link from u to v comes up, nothing special is done! 


Here is an intuitive explanation of why three phases are used in the AAG protocol. 
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Each execution a of the reset protocol can be used to induce work intervals at 
nodes. Let us call a work interval at a node wu, a maximal subsequence of a during 
which node u is not in Ready mode. Thus from the point of view of node u, each 
execution a can be divided into work intervals followed by Ready intervals (intervals 
during which u is in Ready mode.) Now each work interval at node wu can be considered 
to be “caused” by an ABORT packet from a neighbor v (in which case we say that the 
work interval at u has as its parent a work interval at v) or by a reset request (in 
which case, we say that the work interval at u has no parent). Thus, starting from an 


execution a, we can assign each work interval at each node to a tree of work intervals. 


We claim that each tree of work intervals has a height of at most n, where n is 
the number of nodes in the graph. This follows if we can show that any work interval 
tree can contain at most one work interval from any given node. This brings us to 
the crucial observation. Consider any work interval tree T in a. Let J, be the work 
interval corresponding to the root. The use of three phases guarantees us that there 
is some state in I, that is contained in all the work intervals contained in T. This is 
the state in which r sends READY packets to all its children. But that implies that a 
given node wu cannot have two distinct work intervals in TJ’ because these two disjoint 


work intervals do not share a common state. Hence the height of T is at most n. 


Thus, each root interval can “cause” at most one work interval at any node u. But 
each root interval corresponds to either a reset request (or to a link failure in the case 
of dynamic networks). Thus the number of work intervals at a node is at most the 
number of reset requests (plus the number of topology changes in the case of dynamic 
networks). Thus if the number of reset requests (and topology changes) is finite, there 
will only be a finite set of work intervals at each node. Next, it is possible to show 
that each work interval terminates in O(n) time by using induction on the height of 
the abort tree. We combine the last two observations to prove the causality property 
— in finite time after all reset requests (and topology changes) stop, all nodes return 


to Ready mode. 


It is interesting to return to the simple reset protocol (SRP). Consider an execution 
a of SRP that begins in a “bad state” as shown in Figure 7.5. Suppose we construct 
work interval trees from execution a. The result is that we get a single work interval 
tree of infinite height. Each work interval ends in finite time but there are an infinite 


number of work intervals at each node! 
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C.2 Overview of the changes required for stabiliza- 


tion 


In a non-stabilizing setting (e.g. [AAG87], buffers like buffer, |v] are modelled by 
unbounded queues. However, just as in the case of links, stabilizing reset protocols 
must use bounded size queues if they are to stabilize in bounded time. Our solution 
is for v to use a single buffer to store messages from wu, and to require that v sends a 
special &: — Ack packet whenever it removes a message from the buffer. This special 


packet is not needed in the original AAG protocol that uses unbounded queues. 


The first step in making this protocol stabilizing is to make it locally checkable. A 
clear problem with the AAG protocol is that it will deadlock if in the initial state some 
parent edges form a cycle. As in stabilizing spanning tree algorithms [AK Y90, AG90], 
we mend this flaw by maintaining a distance variable at each node, such that a node’s 
distance is one greater than that of its parent. Specifically, distance is initialized to 0 
upon reset request, and its accumulated value is appended to the abort packets. Thus 


we encode an abort packet as a tuple (ABORT, d), where d is a distance. 


Next, we list all the local predicates that are necessary to ensure correct operation 
of the Reset protocol. Note that a significant advantage of our approach is that all we 
have to do is to prove that all local predicates eventually hold. Once we do that we can 
rely on the correctness of the original protocol. However, since a rigorous correctness 
argument for the original protocol did not exist (as far as we knew), we produced a 


correctness argument anyway. 


To ensure that the local predicates are closed predicates, we use the heuristic of 
removing unexpected packet transitions. Recall from Chapter 6, that to do this we 
add checks before processing any packet arriving at the node. We check whether this 
packet could have possibly been sent when the link subsystem (on which the packet 
arrived) is in a good state. Two checks we have added (which were not needed in the 
AAG protocol) are: when an ACK arrives, a node checks whether the ACK expected 
before processing it; when a READY packet arrives, a node accepts it only if it is in 


Converge mode and the packet has come from the node’s parent. 


The use of the distance variable introduces a new problem. Since the distance field 
has a maximum value, say n’, we have to consider the case of a node u that receives an 


(ABORT, n’) packet from neighbor v. If u is in Ready mode, u cannot simply accept v 
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as its parent and set its own distance to be one greater than n’ since n’ is the maximum 
value. Instead, in our code, u pretends that it has received a reset request before the 
(ABORT, n’) packet. Thus u will first become a root and send ABORT packets to all 
its neighbors; then it will send back an ack to v. We call this a spurious reset request 


action. 


Since the new action we have added is just a combination of two existing actions 
in the original protocol, the new action preserves all local predicates and consistency 
conditions of the original protocol. However, it does slightly complicate the proof of 
termination (and hence of the causality condition). Clearly, if the reset protocol can 
keep producing such “spurious reset requests” the protocol may never terminate even 
if all real reset requests stop. Luckily, it is easy to show that within linear time after all 
local predicates of the original protocol hold, no (ABORT, 7’) packets can be received. 
Thus spurious reset requests stop in linear time, and the termination proof is only 


slightly more complicated. 


Next we have to design a local correction action for links, that is taken when a 
violation of the predicates is detected. The main difficulty about designing a correcting 
strategy is making it local, i.,e., to ensure that when we correct a link we do not 
affect the correctness of any other link. An interesting heuristic for this purpose is 
to notice that protocols designed for dynamic networks (like [AAG87]) had to deal 
with link failures and recovery. Now when a link fails and then immediately recovers, 
the original protocol must have established the local predicates for the link that failed 
without affecting the correctness of the other links. Thus the local correction procedure 
we use is essentially identical to the combined code in [AAG87] that is invoked when 


a link fails and recovers. This seems to be a powerful heuristic in general. 


C.3 Mating Relation is not Transitive 


Consider the reset protocol described in Chapter 7 (or the protocol described in 
[AAG87]) when all local predicates hold. We use the scenario shown in Figure C.1 
to show that the mating relation between signal intervals at neighboring nodes is not 
transitive. Thus the reset protocol may cause inconsistent states of the user protocol 
during initial signal intervals. However, the mating relation between final intervals 


is indeed transitive. The original paper [AAG87] showed an example in which the 
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Figure C.1: Counterexample to Show that Mating Relation is not Transitive 


mating relation was not transitive; however, the original paper used an optimization 


(in which ABORT packets were only sent on edges on which messages had either been 


sent or received in the last signal interval). Thus it was not clear whether the problem 


was due to the optimization. In fact, the simple counterexample in |[AAG87] does not 


work as soon as we remove the optimization. 


However, we show in Figure C.1 that there is a (more complicated) counterexample. 


In Figure C.1 there are three nodes A, B and C' that are connected in a cycle 


(not shown). The vertical axis represents time; time increases as we go downwards. 


Because this is a cycle, there is a link between A and C. When A sends a packet to 


C’ (or vice versa), we show this by showing an arrow leaving A and going off the left 
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of the page. We then depict the packet receipt at C’ by an arrow coming in to C from 
the right end of the page. Thus in the second event at C, C sends an ABORT packet 


to A which is received as the second event at A. 


In the initial state, mode(A) = Converge and parent, = B. Thus A is waiting for 
a READY message from B that is sent out at the start of the execution. Assume that 
after B has sends out the READY packet, B does a SIGNAL event immediately and so 
mode(B) = Ready at the start of the execution. We assume that mode(C’) = Ready 
and that there is no other packet in transit in the initial state. Clearly this is a valid 


initial state. 


The first event at C’ is that C sends a user message (say ml) which is received 
by B and then delivered. Immediately after this B sends a user message m2 to A. 
Shortly after this, a reset request occurs at both B and C’. This causes B to send an 
ABORT packet to A and C’, and C to send an ABORT packet to A and B. The ABORT 
packet sent from C’ to A arrives before the READY packet arrives at A. Thus A will 
send an ACK back. Similarly B sends an ACK immediately back to C. The net result 
is that by the time A receives its READY packet, C’ is already in READY mode. We 
also assume C’ has done a signal event immediately after going to READY mode. Next 
the message m2 sent by B arrives at A and C’ sends a third user message m3 which is 
received by A. All this happens before the ABORT packet in transit from B arrives at 
A. 


The net result is as follows. C’ sends two messages ml and m3 in two different 
signal intervals at C’, call them Sc and SG. Message ml is received in a signal interval 
say Sp at B. Node B then sends a message m2 in signal interval Sg that is received 
in a signal interval (say S4 at A.) Finally the message sent by C in S¢ is also received 


in Sa. 


By the definition of the mating relation, messages can only be received from a 
unique mate at a neighbor. Thus Sc mates to Sg and Sg mates to Sy. Also S4 mates 


to Sy. If the mating relation were transitive, Sc would mate to SQ which is impossible. 


The significance of this counterexample is that arguments like “making a reset 
request guarantees a fresh start of the application protocol” do not work. If the 
application is doing any general form of checking, it may detect an inconsistent state 
and keep making reset requests, leading to non-termination. For example, if we were 


to use the global snapshot protocol due to [KP90] to check the application and then 
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use our stabilizing reset, the protocol may never terminate! Our stabilizing reset is 
still useful in a number of cases. For instance, (Chapter 8) when used in conjunction 
with a local snapshot protocol that checks for closed local predicates. In any case, 


termination of the resulting protocol requires careful argument. 


Another significant thing about the counterexample is that it shows that our pair- 
wise definition of a mating relation between neighbors is probably the only statement 
that one can make about the non-final intervals of a reset protocol. The original speci- 
fication of [A AG87]| used a state-based specification. It seems hard to specify this weak 
safety property of non-final intervals in terms of states as opposed to using an external 
behavior specification. In either, case [AAG87] only needed to specify the behavior 
in the final interval. For stabilizing applications, we must specify the behaviors in 


non-final intervals. 


It is interesting to note that the reset protocol of Arora and Gouda [AG90] (after 
adaptation to a message passing model) is likely to ensure a transitive mating relation. 
This is because it does a reset protocol on a tree and only the root (effectively) sends 
out the quivalent of ABORT packets. Of course, our stabilizing protocol can be first 
used to construct a spanning tree, after which we do another reset which is guaranteed 


to work from a single root outwards. 
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Appendix D 


Proofs for Reset Protocol in 


Chapter 7 


D.1 Proving that the Local Predicates of the Reset 


Protocol are Closed 


Recall the definition of L,,, in Definition 7.6.3. 


Our strategy for showing that L,,,, is closed is as follows. If we had to consider every 
action we would have a large number of cases to consider. However, each transition 
only affects a small number of variables. Thus we first isolate the transitions that can 


affect each variable. This reduces the number of cases we have to consider. 


First, we need to define the key transitions. Recall that the code in Figure 7.7 has 
certain code paths marked as VR, VA, DA, IA, FA, RA, and RR. We now define 
these more precisely. It is helpful to refer to the code of Figure 7.7 in understanding 


the following definitions. 


A informal description of these transitions is as follows. A VR (for Valid Request) 
transition is a reset request that causes a node to change its mode to Abort. A VA (for 
Valid Abort) transition is the receipt of an (ABORT, d) packet with d <n’ that causes 
a node to change its mode to Abort. A DA (for Distance Invalid Abort) transition is 
the receipt of an (ABORT, n’) packet that causes a node to change its mode to Abort. 


301 


An IA (for Invalid Abort) transition is the receipt of an (ABORT, *) packet that 
does not cause a node to change its mode to Abort. A FA (for Final Ack) transition 
is the receipt of an ACK packet that causes say node u to send an ACK packet to its 
parent. It is not hard to see that the ack that was received must have been the last 
ack that node u was waiting for. A RA (for Root Ack) transition is the receipt of an 
ACK packet at a root node that causes the root node to change its mode to Ready. A 
RR (for Regular Ready) transition is the receipt of an (READY) packet at a node that 


causes the node to change its mode to Ready. More carefully: 


Definition D.1.1 We call a transition (s,a,s') of R: 


e A VR transition at u if a = REQUEST, and s.mode(u) = Ready. 


e A VA transition at u if a = RECEIVE,,,(ABORT,d) and d <n’ and s.mode(u) = 
Ready. 


e A VA transition at u with respect to v if a = RECEIVE,,(ABORT, d) and s.mode(u) = 
Ready and d < n'. 


e A DA transition atu with respect to v if a = RECEIVE,,,( ABORT, d) and s.mode(u) = 
Ready and d=n’'. 


e AnJA transition at u with respect to v if a = RECEIVE,,,( ABORT, d) and s.mode(u) # 
Ready. 


e A FA transition at u with respect to v if a = RECEIVE.,,( ACK) and s.mode(u) = 


Abort and s.parent, = v and s'.mode(u) = Converge. 


e A RA transition at u if a = RECEIVE,.(ACK) and s.mode(u) = Abort and 
s'.mode(u) = Ready. 


e A RR transition at u if a = RECEIVE.,,( READY) and s.mode(u) # Ready and 
s'.mode(u) = Ready. 


In the following lemma we will say that a Boolean condition b is established by some 
transition (s,a,s’) if b is false in s but true in s’. Recall that we wish to prove that 
each L,,,, is closed. The next lemma makes this job easier, by isolating the transitions 


that can establish various Boolean conditions used in the definition of L,,,. 
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We start with the observation that we do not need to consider the transition DA 
explicitly because a DA transition at u with respect to v can be simulated by two other 
transitions: first a VR transition at u followed immediately by an JA transition at u 
with respect to v. Thus in the following lemmas and proofs we assume that the DA 


transition does not exist, 


Lemma D.1.2 


1. ack,|v| = true can only be established by a VR or a VA transition at u. 
2. Al(u,v) = true can only be established by a VR or a VA transition at u. 
3. A2(u,v) = true can only be established by a VA transition at v with respect to u. 


4. A3(u,v) = true can only be established by a IA or a FA transition at v with 


respect to wu. 
5, A2(u,v) = false can only be established by a FA transition at v with respect to wu. 


6. ack,|v| = false can only be established by a transition (s,a,s') such that s.ack,|[v] = 
true and a = RECEIVE,,,(ACK). 


7. parent, =v can only be established by a VA transition at u with respect to v. 


8. mode(u) = Converge can only be established by a FA transition at u with respect 


to some neighbor zx. 
9. mode(u) # Converge can only be established by a RR transition at u. 


10. Cl(u,v) = true can only be established by a IA or a FA transition at u with 


respect to v. 


11. C2(u,v) = true can only be established by a transition (s,a,s') such that s.ack,|u] = 
true and a = RECEIVE,,,(ACK). 


12. C3(u,v) = true can only be established by a RR or RA transition at v. 


13. C2(u,v) = false can only be established by a RR or RA transition at v. 
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Proof: By inspection of the code. J 


Notice that in the code we often enqueue packets to queue, |v]. But since queue,|v| 
is finite (it only has room for 5 packets), this allows the possibility of a transition 
(s, 7,5’) causing a packet to be dropped if queue,|v] is full in the previous state. The 
next lemma shows that packets will not be dropped if L,,, holds in s. We will tacitly 


assume this lemma in what follows without making explicit reference to it. 


Lemma D.1.3 For any leader edge (u,v) and any transition (s,a,s') of R, ifs € Lu». 
then in s', Q holds. Also if as part of the code for a, a packet p is enqueued on queue,,|v| 
then p will be added to the tail of queue,,|v] in s'. 


Proof: To show Q, we show that when a packet of a certain type is in zqueue,|v], 
then no action will enqueue a packet of the same type. The five types of packets to 
consider are (ABORT, *) packets, ACK packets, READY packets, © — ACK packets and 
&i-messages. The second part also follows from this claim because zqueue, |v] has room 


for five packets. 


Suppose there is an (ABORT, *) packet in zqueue,|v] in s. Then (by A) s.mode(u) = 
Abort and hence this cannot be a VA or VR transitions at u. But these are the only 
transitions that can enqueue another (ABORT, *) packet to equeue,[v] (see Lemma D.1.2, 
item 2). 


Suppose there is an ACK packet in zqueue,|[v] in s. Then, (by B) Al(v,u) = false 
and A2(v,u) = false in s. Thus (by Lemma D.1.2, item 4), this cannot be a transition 


that can enqueue another ACK packet to zqueue,,|v]. 


Suppose there is a READY packet in zqueue,,|v] in s. Then (by G) either s.mode(u) = 
Ready or there is an ABORT in zqueue,,|v| after the READY packet. In the former case, 
(by Lemma D.1.2, item 12) this this cannot be a transition that can enqueue another 
READY packet to zqueue,|v]. In the latter case, (by A) s.ack,|v] = true, and (by B) 
A3(u,v) is false in s'. Thus again (by Lemma D.1.2, item 12) this this cannot be a 


transition that can enqueue another READY packet to zqueue,,|v]. 


Suppose there is a ) message in zqueue,,|v] ins. Then (by #1) in s, freem,|v] = false. 


Thus even if a = SENDM,,,(m), the code will not enqueue m on zqueue,|v]. 


Suppose there is a ) — ACK message in zqueue,|v] in s. Then (by 1), buffer,,|v| 


must be empty and so the code cannot queue a » — ACK on zqueue, |v]. Il 
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We now proceed to prove that each of the predicates from A to H are closed in a 
series of four lemmas: Lemma D.1.4, Lemma D.1.5, Lemma D.1.6, Lemma D.1.7, and 


Lemma D.1.8. 


Lemma D.1.4 For any leader edge (u,v) and any transition (s,a,s') of R, ifs € Lu». 
then in s', A and B hold. 


Proof: We consider four cases: 


e Suppose ack,|v] = false in s but ack,|v] = true in s’. Then (by A) Al(u,v), 
A2(u,v), and A3(u,v) are false in s. Also by Lemma D.1.2, the transition must 
be a VA or a VR transition at u which causes Al(u,v) to become true and leaves 


A2(u,v) and A3(u,v) as false in s’. 


e Suppose ack,|v| = false in s and s’. Then (by A) Al(u,v), A2(u,v), and A3(u, v) 
are false in s. Also by Lemma D.1.2, items 1 and 2, Al(u,v) cannot hold in s’ 
without making ack,|v] = true hold in s’. Also by Lemma D.1.2, item 3, A2(u, v) 
cannot become true in s’ if Al(u,v) is false in s. Also by Lemma D.1.2, item 4, 
A3(u,v) cannot become true in s’ if Al(u,v) and A2(u,v) are false in s. Thus 
Al(u,v), A2(u,v), and A3(u,v) are false in s’. 


e Suppose ack, |v] = true in s but ack,|v|] = false in s'. By Lemma D.1.2, item 6, 
a = RECEIVE,,,(ACK) and so A3(u,v) is true in s. By B, Al(u,v) and A2(u,v) 
are false in s. Also, (by Q) there is exactly one ACK packet in s.zqueue,[u] and 
hence Al(u,v), A2(u,v), and A3(u,v) are false in s’. 


e Suppose ack, |v] = true in s and s’. Then (by A) exactly one of Al(u,v),A2(u, v), 
or A3(u,v) is true in s. If Al(u,v) is true in s, then Al(u,v) becomes false 
in s’ (by Lemma D.1.2, items 3 and 4) iff exactly one of A2(u,v) or A3(u,v) 
becomes true in s’. If A2(u,v) is true in s, then A2(u,v) becomes false in s' (by 
Lemma D.1.2, item 5) iff A3(u,v) becomes true in s’. Finally if A3(u,v) is true 
in s, then A3(u,v) cannot become false in s’ (by Lemma D.1.2, item 6) without 


causing ack,|v| = false in s', a contradiction. 
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Lemma D.1.5 For any leader edge (u,v) and any transition (s,a,s') of R, ifs € Luy, 


then C and D are true for (u,v) in s’. 


Proof: We consider cases: 


e Suppose parent, # vin s'. ThenC and D hold trivially in s’. Suppose parent, # v 
in s and parent, = v in s'. Then by Lemma D.1.2, item 7, this must be a VA 
transition at u with respect to v. Thus s’.mode(u) = Abort, Thus Al(v,u) is 
true in s and hence (by A) A3(v,u) = false in s and hence C'l(u,v) is false in 
s and hence in s’. Also, (by A) ack,|u] = true in s and s’ and hence ('2(u,v) is 
false in s’. Next (by G and Q) we can infer that C'3(u,v) is false in s and hence 
in s’. (This is because if C3(u,v) were true in s then (by G) there must be a 
second (ABORT, *) packet in zqueue,|u] which would violate Q.) 


Thus in the remaining cases we assume that parent, = v in s and s’. 


e Suppose mode(u) # Converge in s and s’. Then Cl(u,v), C2(u,v), and C3(u, v) 
are false in s. Also by Lemma D.1.2, item 10, Cl(u,v) can become true in s’ 
only if mode(u) = Converge in s'. | By Lemma D.1.2, item 11, C'2(u,v) can 
become true in s’ only if Cl(u,v) is true in s. Finally by Lemma D.1.2, item 


12, C3(u,v) can become true in s’ only if Cl(u,v) or C2(u,v) is true in s. Thus 


Cl(u,v), C2(u,v), and C3(u,v) are false in s’. 


e Suppose mode(u) # Converge in s and mode(u) = Converge in s’. Then (by C) 
Cl(u,v), C2(u,v), and C'3(u,v) are false in s. Also by Lemma D.1.2, item 8, 
ais a FA transition at wu with respect to v. Thus after this transition, C1(u, v) 


becomes true in s', and C'2(u,v) and C’3(u,v) remain false in s’. 


e Suppose mode(u) = Converge in s and mode(u) # Converge in s'. Then by 
Lemma D.1.2, item 9, a is a RR transition at u. Thus C3(u,v) is true in s; 
hence (by D) Cl1(u,v) and C'2(u,v) are false in s and s’. Also, (by Q) there 
is exactly one READY packet in zqueue,|u] in s and this is removed after the 


transition, and so (’3(u,v) is false in s’. 


1Note that this cannot be an JA transition at u with respect to v because otherwise s.mode(u) = 


Abort which together with s.parent, = v would violate A. 
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e Suppose mode(u) = Converge in s and s’. Then (by C and D) exactly one of 
Cl1(u,v),C2(u,v), or C3(u,v) is true in s. First suppose Cl(u,v) is true in s. 
Then (by A) ack,[u] = true in s. Thus Cl(u,v) becomes false in s’ iff exactly 
one of C'2(u,v) or C3(u,v) becomes true in s’. Also if Cl(u,v) remains true in 
s’, then ack,|w] = true in s’ by Lemma D.1.2, item 6. Thus by Lemma D.1.2, 


items 11 and 12, C2(u,v) and C3(u,v) cannot become true in s’. 


Next, notice that since we have assumed that mode(u) = Converge in s, C1(u,v) 
remains false in s' (by Lemma D.1.2, item 10) if it is false in s. Suppose C'2(u, v) 
is true in s. Then C'2(u,v) becomes false in s’ (by Lemma D.1.2, item 13) iff 


C’3(u,v) becomes true in s’. 


Finally if C'3(u,v) is true in s, then C3(u,v) cannot become false in s’ without 
causing mode(w) = Ready in s’, a contradiction. Also by Lemma D.1.2, item 11 


C'2(u,v) cannot become true in s’ since C1(u,v) is false in s. 


Lemma D.1.6 For any leader edge (u,v) and any transition (s,a,s') of R, ifs € Luy, 


then E and F are true for (u,v) in s'. 


Proof: First consider F. Suppose there is no (ABORT,*) packet in zqueue,|v] in 
s. Then if an (ABORT,*) packet is in zqueue,|v] in s’, then this must be a (by 
Lemma D.1.2, item 2) VR or VA transition at u. However, after such a transi- 
tion there is an (ABORT,d) packet in zqueue,|v|], where d = dist, + 1. Suppose 
there is a (ABORT, d) packet in zqueue,|v| in s with d = s.dist, + 1. Then (by A) 
s.mode(u) = Abort. Now the only transitions that can change dist, are VR or VA 


transitions at u, but such transitions are not enabled in if s.mode(u) # Ready. 


Now consider € and consider three cases. 


e parent, # v in s’: then € holds trivially. 


e parent, # vin s but parent, = v in s’: Then by Lemma D.1.2, item 7, a must 
be a RECEIVE,,,(ABORT,d) event with s.mode(u) = Ready. But by F in s, 
d= s.dist, + 1. Thus in s’, dist, = dist, +1. and mode(u) # Ready. 
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e parent, = v in s and s’. If in s, C3(u,v) is true, then C3(u,v) can become 
false in s' if a = RECEIVE,,(READY), but in that case s’.parent, = nil. If in 
8, dist, = dist, + 1 and mode(v) # Ready, then this transition cannot be a VR 
or VA transition at v or u. But only such transitions can change either dist, or 
dist,. Also if s'.mode(v) = Ready, then this must be a RR or RA transition at v 


which results in C3(u,v) becoming true in s’. 


Lemma D.1.7 For any leader edge (u,v) and any transition (s,a,s') of R, ifs € Lu», 


then G is true for (u,v) in s’. 


Proof: Suppose in s there is no p in zqueue,|v] such that p is either a READY packet 
or p is a 4-message or p is a & — ACK. Then if there is such a packet in s’, then, by 
the code, s.mode(v) = Ready and we are done. Suppose in s there is a p in zqueue,|v| 
such that p is either a READY packet or p is a }-message or p is a ©) — ACK. Then by 
G either there is an (ABORT, *) packet in zqueue,,|v] or s.mode(u) = Ready. Consider 
the first case. Since the channel from u to v is FIFO, the (ABORT, *) packet cannot 
be removed from zqueue,|v] in s’ without also removing packet p. Consider the second 
case. Then if s’.mode(v) # Ready then this transition must be a VA or VR transition 
at u which would result in adding an (ABORT, *) packet to the end of zqueue,|v] in s’. 


Lemma D.1.8 For any leader edge (u,v) and any transition (s,a,s') of R, ifs € Lu», 


then H is true for (u,v) in s’. 


Proof: We consider all the actions a that can affect this predicate. For each action 


considered, a symmetrical argument holds for the action with u and v interchanged. 


If a = SENDM,,,(m) and s.freem,|v| = false, then message m is dropped and there 
is no change to the concerned variables. If a = SENDM,,,(m) and s.freem,|v] = true, 
then there is no X-message in M,, in s and no & — ACK in s.zqueue,|u]. Also, (by 
Qand H) there is at most one (ABORT, *), ACK, or READY packet in quewe,[v] in s. 
Since queue, |v] can store five packets, after this event m is placed in queue,|v] and 


s'.freem,,|v| = false and and there is no © — ACK in s.zqueue,|w]. 
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If a = RECEIVE,,.(m), m € &, then m is enqueued in buffer,|u] and none of 
the concerned variables change. Note that by H, buffer,|u] is empty in s. If a = 
RECEIVE,,,(ABORT, *) and s.buffer,[u| is empty in s, then there is no change to the 
concerned variables. Suppose a = RECEIVE,,,(ABORT,*) and s.buffer,|u] contains a 
message m in s or a = RECEIVEM,,(m). Then (by H) in s, freem,|v] = false, there 
is no & — ACK in aqueue,|u], and no other message in M,,, besides m in buffer, |u]. 
Then in s’, all variables remain unchanged except that buffer,|u] becomes empty and 
a & — ACK is added to s.zqueue,|w]. 


Similarly, if a = RECEIVE,,,(% — ACK), then by Hin s, freem,|v| = false, there is 
no & message in M,, and exactly one & — ACK in aqueue,|u]. The result is that the 


& — ACK is removed from zqueue,|u] and freem,,|[v| becomes true in s’. JJ 


D.2 Proving that the Reset Protocol Behaviors are 


Timely, Causal, and Consistent 


We will show that every behavior of R|L is timely and causal and satisfies the consis- 
tency property. We will do so in the next five subsections. First, we prove a series of 
useful preliminary lemmas. In the second subsection, we prove that every behavior is 
timely. In the third subsection, we prove that every behavior satisfies the consistency 
property. In the fourth subsection, we prove that every behavior is causal. Finally, we 


tie everything together and show that every behavior @ of R|L is in RP. 


We will assume this claim implicitly in what follows. 


D.2.1 Useful Claims and Lemmas for Reset Protocol 


The first lemma is the important Termination Lemma that states that the mode will 


become Ready in O(n) time. 


Lemma D.2.1 Termination Lemma Consider an execution a = 589,41,51,... of 
R\|L. For any state s; there is another state s; that occurs within O(n) time after s; 
and such that s;.mode(u) = Ready. 


309 


Proof: A formal argument can be made based on the intuitive “proof” given in 
Section 7.7.1. We omit it here. J 


The next lemma is the Signal Lemma. The lemma states that once the status of a 
node u is off (recall that this means that either signalbit, = true or mode, # Ready) 


then a SIGNAL, event is guaranteed to occur in linear time. 


Lemma D.2.2 Signal Lemma: For any ezecution a of R\L, if s;.status(u) = off 


then a SIGNAL, event occurs within O(n) time after s;. 


Proof: Suppose s;.mode(u) = Ready and s;.signalbit, = true. Then it is easy to see 
from the code that a SIGNAL, action is enabled and will remain enabled until it occurs 


in constant time after s;. 


The only other possibility is that s;.mode(u) # Ready. Let s, be the first state 
after s; such that s,.mode(u) = Ready. We know from Lemma D.2.1 that such a 
state exists and s, occurs within O(n) time after s;. Then s,_1.mode(u) # Ready and 
s,.mode(u) = Ready. Thus from the code it is easy to see that s,.signalbit = true. 
Now we are back to the first case and hence a SIGNAL, event must occur in constant 


time after s,. Jf 


The next claim is a variation of the Signal Lemma. It states that the status of a 


node wu cannot change from off to on until the next SIGNAL, event. 


Claim D.2.3 For any execution a of R\L, if s;.status(w) = off and s,.status = on 


and k > 1, then there is a SIGNAL, event between s; and s, ina. 


Proof: From the code, the only way status(u) can change from off to on is by a SIGNAL, 
event. Note that any action that changes mode(u) to Ready also sets signalbit, to true, 


which leaves status(u) unchanged. J 


Next we show that any packet queued on the outbound queue for a link will be 
delivered in constant time. This follows from the fact that the outbound queue has a 


size of at most 4. 


Claim D.2.4 Consider any pair of neighbors u,v and any execution a. If there is a 
packet p in zqueue,|v| in some state s; of a, then in constant time after s; there is a 
RECEIVE,» (p) event. (i.e., any packet in either the outbound queue for a link or on 


the link itself is delivered within constant time.) 
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Proof: We know from Lemma 5.6.6 that the packet at the head of queue,,[v] is placed in 
Qu, (i.e., is placed in the channel) in ¢, time and that any packet in Q,,, is delivered in 
t; time. The lemma follows since queue,,|v| has a size of at most 4, and t, is a constant. 


Next, we show that within constant time the signalbit variable at a node becomes 


false. 


Claim D.2.5 Consider any node u,v and any execution a. If signalbit, = true in 
some state s; of a, then in constant time after s; there 1s a state in which signalbit, = 


false. 


Proof: This follows because if signalbit, = true in s;, the SIGNAL, event is enabled 
and signalbit, will remain true unless the SIGNAL, event occurs (see code). Thus since 
each SIGNAL, action is in a separate class, a SIGNAL, action will occur in constant 


time after s;, resulting in a state (see code) in which signalbit, = false. I 


The next claim (which is used to show the consistency property) states the follow- 
ing. Suppose there is some interval in which the mode of a node u is Ready at the start 
and end of the interval but is not Ready somewhere within the interval. Consider any 
neighbor v of u. Then there must be some point within the interval during which an 
ABORT packet arrives at v; also at this point any messages in transit from u to v have 


been “flushed” out. 


Claim D.2.6 Consider any pair of neighbors u,v and any execution a of R|L. Con- 
sider any three states s;, s; and s, in a such thati <j <k and s;.mode(u) = Ready, 
s;.mode(u) # Ready and s,.mode(u) = Ready. Then there is some j' such that 
i <j' <k and a; is a RECEIVE,,(ABORT,*) action and s;.M,, does not contain 


any -message. 


Proof: Let s; be the first state after s; in which s;.mode(u) # Ready. Such a state 
must exist by hypothesis and it must be that | < 7. Thus by the code, A3(u,v) is 
true in s; (i.e., there is an abort packet in equeue,[v] in s;). Let sj be the first state 
after s; in which A3(u, v) is false (i.e., the first state after s; in which the abort packet 
is delivered). We know from Claim D.2.4 that such a state exists. Also, we know 
from A, that in the interval [s;,5,-], mode(w) # Ready. Thus j’ < k. Also a; must 
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be a RECEIVE,,,(ABORT, *) event and following such an event it is easy to see from 
the code that mode(v) # Ready and that buffer,|u| is empty. Also we know that in 
the interval [s;,s;/], since mode(wu) # Ready no X-message was added to M,,,. Also 
any -message in zqueue, |v] in s; must have been removed from zqueue, |v] before s,) 
because zqueue,|v] is a FIFO queue. Thus s;.M,,, does not contain any U-message. 


The next claim (which is also used to show the consistency property) is a mild 
corollary of the previous claim. Suppose there is some interval in which mode(u) = 
Ready at the start of the interval, and u’s signal bit is false in the interval, and u does 
a signal at the end of the interval. Then the mode of u must be Ready at the end of 
the interval and must have been not Ready somewhere within the interval. Thus the 


previous claim applies, along with its consequences. 


Claim D.2.7 Consider any pair of neighbors u,v and any execution a of R|L. Con- 
sider any state s; such that s;.mode(u) = Ready and another state s,, i’ > 1, such that 
8; .signalbit, = false. Suppose that after s; there is a SIGNAL, action ay. Then there 
is some j where 1 <j < k such that: 


e a; is a RECEIVE,,,( ABORT, *) action. 
e s;.status(v) A on 


e s;.M,, does not contain any U-message. 


Proof: In the state just before a,, signalbit, = true but in s,, signalbit, = false. 
Let s; be the first state before s,_; in which signalbit, = false. Also | > i’ because 
s;.signalbit, = false. Thus from the code it must be that s;.mode(u) # Ready and 
$141.-mode(u) = Ready. The lemma follows by using Claim D.2.6 to the three states s;, 
s; and s;,, and by observing that in any state that follows a RECEIVE,,,( ABORT, *) 
action, mode(v) # Ready and hence status(v) 4 on. If 


Notice that the requirements of the previous claim are satisfied if status(u) = on 
at the start of the interval and there is a signal at u at the end of the interval. We 


state this corollary to the previous claim as a separate claim as it is used often. 
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Claim D.2.8 Consider any pair of neighbors u,v and any execution a of R|L. Con- 
sider any state s; such that s;.status(u) = on. Suppose that after s; there is a SIGNAL, 


action a,. Then there is some j7 where i <7 <k such that: 


e a; is a RECEIVE,,,( ABORT, *) action. 
e s;.status(v) A on 


e s;.M,, does not contain any U-message. 


Proof: Follows from Claim D.2.7. Jj 
The next three claims are all used to prove the timeliness property. 


Consider any two neighboring nodes u and v. The next claim states that if the 
status of both u and v is on for a sufficiently large constant, then u must deliver a free 
event (indicating that u is willing to accept a new message to be sent to v) within this 


time. 


Claim D.2.9 Consider any pair of neighbors u,v and any execution a of R|L. Con- 
sider any state s;. Then in constant time after s; either a FREEM,, occurs or there is 


a state s; such that either s;.status(u) = off or s;.status(v) = off. 


Proof: Suppose that status(u) = on and s;.status(v) = on for c time after s;, where 
cis a large enough constant to make the following argument work. By H, in state s; 


either: 


e freem,,|v| is true. 
e zqueue,|u| contains a % — ACK. 


e M,,, contains a Li-message. 


In the first case, assuming c is large enough, status(u) = on for a constant time 


after s; which causes a FREEM,,, to occur in constant time after s;. 


In the second case, if there is a } — ACK in zqueue,|u] then by Claim D.2.4 within 
constant time a RECEIVE,,,( — ACK) event occurs which causes freem,,|v] to become 


true, leaving us in the first case. 
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For the third case, we can use an argument similar to the second case to show that 
in constant time after s;, a RECEIVE,,,(m) event occurs which causes m to be placed 
in buffer, |u]. Assuming again that c is large enough this implies that in constant time 
after s; either a RECEIVEM,,,(m) event or a RECEIVE,,,(ABORT) event occurs. Either 
of these events will cause a & — ACK to be placed in zqueue,|u], which brings us back 
to Case 2. J 


Consider any two neighboring nodes u and v. The next claim states that if the 
status of both u and v is on for a sufficiently large constant after a message is sent 


from u to v, then the message will be delivered to v within this time. 


Claim D.2.10 Consider any pair of neighbors u,v and any execution a of RIL.. 
Consider any safe SENDM,y.(m) action in a. Then in constant time after this action 
either a RECEIVEM,.»(m) occurs or there is a state s; such that either s;.status(u) = 
off or s;.status(v) = off. 


Proof: Similar to proof of Claim D.2.9. Let us denote the safe SENDM,,,(m) event by 
a;. Suppose that status(u) = on and s,.status(v) = on for c time after aj, where cis a 
large enough constant to make the following argument work. Then since a, is a safe 
send it is easy to see from the code that freem,|v] is true in the state before a;. Also 
by our assumption, status(w) = on in the state after a;. So m is placed in queue,|v| 
in the state after a;. Thus by Claim D.2.4, in constant time after a;, m is placed in 
buffer,[u| and if status(v) = on for constant time after this, a RECEIVEM,,,(m) event 


occurs. Jf 


D.2.2 Every Behavior of R|L is timely 


We prove that every behavior @ of R|L is timely by showing each of the four properties 


in Definition 7.3.3. Corresponding to the four properties, we have four lemmas. 


We will leave the first property of a timely behavior (i.e., that all messages received 


after O(n) time are normal) to the end of this section. We start by showing the second 


property. 


Lemma D.2.11 Periodic Free Events: Consider any pair of neighbors u,v and 


any ezecution a of R|L and any state s; ina. Then either a FREEM,,»(m) occurs 
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in constant time after s; or a SIGNAL, action occurs in O(n) time after s; or or a 


SIGNAL, action occurs in O(n) time after s;. 


Proof: We know from Claim D.2.9 that for some constant c, in c time after s; a 
FREEM,,(m) occurs or there is a state s, such that either s,.status(u) = off or 
s,.status(v) = off. In the first case, we are done. In the second case, we know from 
the Signal Lemma (Lemma D.2.2), that within O(n) time after s, either a SIGNAL, 


or a SIGNAL, event occurs, Jf 


Next, we prove that R|L satisfies the third property of a timely behavior. 


Lemma D.2.12 Timely Message Delivery: Consider any pair of neighbors u,v 
and any execution a of R|L. Consider any a; that is a safe SENDM,.(m) action in a. 
Then either a RECEIVEM,,.(m) occurs in constant time after a; or a SIGNAL, action 


occurs in O(n) time after a; or or a SIGNAL, action occurs in O(n) time after a;. 


Proof: We know from Claim D.2.10 that for some constant c, in c time after a; a 
RECEIVEM,,,(m) occurs or there is a state s; such that either s;.status(u) = off or 
s;.status(v) = off. In the first case, we are done. In the second case, we know from 
the Signal Lemma (Lemma D.2.2) that within O(n) time after s; either a SIGNAL, or 


a SIGNAL, event occurs, J 


Next, we prove that R|L satisfies the fourth property of a timely behavior. 


Lemma D.2.13 Signals at a Node induce Signals at Neighbors: Consider any 
pair of neighbors u,v and any execution a of R|L. There is some constant c such that 
for every SIGNAL, event a; that occurs at time greater than a.start + c-n there is a 


SIGNAL, event that occurs in linear time before or after a;. 


Proof: First, within linear time of the start of 8, there must be some state s, in which 
mode(u) = Ready (by the Termination Lemma). In constant time after s,, there must 
be some state s; in which signalbit, = false by Claim D.2.5. Consider any SIGNAL, 
action a; that occurs after state s;. In the state before a,;, segnalbit, = true but in state 
s;, signalbit,, = false. Consider the first state s, before s; in which szgnalbit,, = false. 


By Claim D.2.5, s; occurs in constant time after s,. Also since 5,4,.signalbit, = true, 
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the code tells us that that s,.mode(u) # ready. But we know that s,.mode(u) = 
Ready. Consider the first state s; before s, in which mode(u) = Ready. By the 


Termination Lemma, s; occurs in linear time before s,. 


Thus we have identified two states s; and s,, with k > [ such that both states occur 
before the SIGNAL, event a; and such that s;mode(u) = Ready and s,.signalbit, = false. 
By applying Claim D.2.7 to s;, s, and a; we know that there is some state s,, in the 
interval |s;,s;] in which status(v) # on. Thus by the Signal Lemma, a SIGNAL, event 
occurs within linear time after s,,. But since s; occurs within linear time before s,, 


the SIGNAL, event occurs in linear time before or after a;. Il 


Finally, we prove that R|L satisfies the first property of a timely behavior. This 


requires more work and so we start with two claims. The claims are quite intuitive. 


The first claim states that in any suffix of an execution, all except possibly the first 


packet received on a link are normal packets that have been sent in this execution. 


Claim D.2.14 Consider any pair of neighbors u,v and any execution a of R|L. Let 
8; be the first state in a such that there is no Xi-message in s;.M,,. Consider any a, 
that is a RECEIVE,,,(m) event that occurs after s; in a. Then: 


e There is a SENDu»(m) action a; before a, and such that there are no RECEIVE,» (*) 
events in between a; and ay. We will call the earliest such a; the send correspond- 


ing to a, ina. 


e In state s; (i.e., the state immediately after a; in a), status(u) = on and in state 


Sz, status(v) = on 


e In all states in the interval [s;, 5,1], m is in My». 


Proof: It is clear from the code that in s,_1, m must be in buffer,,|v] and hence m is 
in M,,,. But since in s;, M, is empty, there must be a SEND,,,(m) action between s; 
and s,_; which added m to M,,,. Let a; be the first such action that occurs before a,. 
In the state immediately after a; by the code status(u) = on. Clearly, there cannot 
be a RECEIVE, ,,(*) action between a; and a, because (from 711), M,,, contains at one 
most one message in any state. But a RECEIVE,,,(m) action is the only action that 
can remove m from M,,, and hence m is in M,,, in the interval [s;,s,_1]. Also from 


the code, in the state immediately after a RECEIVE,,,(m) action, status(v) = on. JJ 
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Claim D.2.15 Consider any pair of neighbors u,v and any execution a of R\L and 
any state s; ina. Then there is some state 8m, that occurs in O(n) time after s; such 


that there is no Ui-message in 8y.M,,y- 


Proof: From Lemma D.2.1, there is a state s; that occurs in O(n) time after s; such 
that s;.mode(v) = Ready. From Claim D.2.5, in constant time after s; there is a state 


sj in which signalbit,, = false. 


From Lemma D.2.1, there is a state s, that occurs in O(n) time after sj such that 
s,.mode(u) = Ready. From Claim D.2.5, in constant time after s, there is a state s,) 


in which szgnalbit,, = false. 


From Lemma D.2.11, some event a; occurs within O(n) time after s,, where a; is 
either a FREEM,,, event or SIGNAL, or a SIGNAL, event. In the first case, we are done 
by predicate 7 which shows that FREEM,,, cannot occur unless M,,,, is empty. In the 
second case a; is a SIGNAL, event, then by Claim D.2.7, there is some state s; in the 
interval [s,, s;] such that there is no X-messagein s,.M,,y. If a; is a SIGNAL, event, then 
by Claim D.2.7 there is a state say sy; in the interval [s;, s;] such that s;.status(w) = off. 
Thus by the Signal Lemma (Lemma D.2.2), a SIGNAL, event occurs within O(n) time 


after sy, which brings us back to the second case. J 


We can now show the first part of the consistency property (see Definition 7.3.5), 
that every message received after O(n) time is normal; it is almost immediate from 


the last two claims. 


Lemma D.2.16 Normal Receipt of Messages: Consider any any execution a of 
R|L and any suffiz y of a with first state 59. There is some constant c such that every 
every receive event that occurs at time greater than sy.time+c-n ina is normal. Also 
if a; 1s any normal receive event and a; is the send corresponding to a;, then a; occurs 


within O(n) time after a;. 


Proof: Consider any pair of neighbors u,v. By Claim D.2.15, there is some s; that 
occurs in O(n) time after s9 and such that there is no L-message in s;.M,,,. Let us call 
j the quiescent index for link (u,v). Further, let k be the largest quiescent index over 
all possible links (u,v). Clearly s, occurs in O(n) time after s9. Also by Claim D.2.14, 


all receive events that occur after s, in y are normal. 
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Also let a; be any normal receive event and a; be the send corresponding to a;. By 
Claim D.2.14, in all states in the interval [s;,s;_1], mis in M,,. But by Claim D.2.15, 
there is some state s, that occurs in O(n) time after s; such that there is no &-message 


in s%.M,,,. Thus a; occurs within O(n) time after a;. I 


And now we come to the main result of this subsection: 
Lemma D.2.17 Every behavior of R|L is timely. 


Proof: Immediate from Definition 7.3.3 and Lemmas D.2.16, D.2.13, D.2.11, and 
D.2.12. J 


D.2.3. Every Behavior of R|L satisfies the consistency prop- 
erty 


We prove that every behavior 8 of R|L satisfies the consistency property by showing 
each of the five properties in Definition 7.3.5. We have the following preliminary claim 


that is essential for proving the consistency property. 


The claim states the following. Consider some interval and some node wu and 
suppose that the status of u is on at the start of the interval and the interval ends 
with uw receiving a normal message m from v. Suppose also that there is a SIGNAL, 
event in the interval. Then from Claim D.2.8 we know that there is some point within 
the interval during which an ABORT packet arrives at v and such that any messages 
in transit from u to v have been “flushed” out. However. this claim goes further and 
states that the point at which the ABORT packet arrives at v occurs before v sends 


message m. 


Claim D.2.18 Consider any pair of neighbors u,v and any execution a of R|L. Con- 
sider any i,j,k such thati <j <k. Suppose s;.status(u) = on, a; is a SIGNAL, event, 
and ay, 1s a normal receive event at u from v. Let a, be the send corresponding to ag. 


Then there is some i <j’ < k’ such that status(v) 4 on and s;.M,, is empty. 


Proof: By Claim D.2.8 applied to s; and a; we know there is some j’ where 1 < j’ <j 
such that: 
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e a; is a RECEIVE,,,(ABORT, *) action. 
e s;.mode(v) # Ready 


e s;.M,, does not contain any L-message. 


As long as we can prove that 7’ < k’ we are done. Suppose not. By A we know that 
s;'.ack,|v| = true. But since ax is a receive action, s,.status(u) = on, and so we know 
that s,.ack,|v] = false. Let s; be the first state after sj; such that s;.ack,|v] = false. 
Thus 7’ <1 < k. But we know from the code that ack,|v| is only set to false after a 
RECEIVE,,,(ACK) event and so a; must be a RECEIVE,,,( ACK) event. But by B we 
know that there is no ACK packet in M,,, in state s;. Thus there must be some n, 
j'<n<k such that a, = SEND,,(ACK). Intuitively, what we have shown is that in 


the interval [s;:,s,] an ack packet must have been sent by v and received by u. 


Now we can obtain the required contradiction. We will only sketch the rest of 
the argument informally. Suppose for contradiction, that 7’ > k'. Then it must be 
that the ack sent (in action a,) was sent after the user message sent (in action ag’). 
But then (essentially because the channels and queues are FIFO), the corresponding 
user message receipt (i,e., action a,) must have occurred before the corresponding ack 
packet receipt (call this action a). Thus k < 1. Also, since 7’ < k, we know that 
k lies in the interval |s,-,s;_1].. But we know from A that in the interval [s,/, s;_1], 
mode(u) # Ready since in this interval wu is still waiting for an ack from v. But this 
contradicts the fact that in a state s, immediately following a receive event such as 


a,, mode(u) must be Ready. I 


Next, we show a lemma (see Figure D.1) which states (in essence) that messages 
sent in a signal interval at v can be received in at most one signal interval at wu; 
conversely messages received in a signal interval at wu could have been sent in at most 
one signal interval at v. This will help establish that each signal interval at u can have 


at most one mated signal interval at v. 


Lemma D.2.19 Send Consistency: Consider any pair of neighbors u,v and any 
execution a of R|L. Let a; and a, be any two normal receive events at u from v in a. 
Let a; and a, be the send events corresponding to a; and a, respectively. Then there 


18 @ SIGNAL, event between a; and a,, iff there is a SIGNAL, event between a; and ag. 
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Figure D.1: Send consistency: there is a signal between the two receives at u iff there is a signal between 


the two corresponding sends at v. 


Proof: Assume without loss of generality that 7 < k. Suppose there is a SIGNAL, 
event, say Gm between a; and am. By Claim D.2.14, s;.status(v) = on. Then by 
Claim D.2.8 there must be some p, | < p < m’, such that s,.status(u) = off and 
M,. is empty. But by Claim D.2.14, M,,, is non-empty in the interval [s;,s;]. Thus 
since p > 1, it must be that p > 7. Also p < m’ and m’ < m and by Claim D.2.14, 
m< k. Sop < k. Thus there is a state s, that occurs in the interval [s;,s,] in 
which status(u) = off. Also we know by Claim D.2.14 that s,.status(u) = on. Thus 


by Claim D.2.3, a SIGNAL, event must occur in the interval [s;, sz]. 


The reverse argument is slightly different. Suppose there is a SIGNAL, event, say a,’ 
between a; and ay. By the code, s;.status(v) = on. Thus by Claim D.2.18, applied to 
$;, 44: and a, we know that there is some 7’ such that 7 <j’ < mand s,:.status(v) 4 on. 
But since | < j, 8; occurs in the interval [s;, 5] and status(v) = off. Also we know 
by Claim D.2.14 that s,,.status(v) = on. Thus by Claim D.2.3, a SIGNAL, event must 


occur in the interval [s;,5,]. I 


Next, we show a second lemma (see Figure D.2) which states (in essence) that a 
signal interval at u cannot send messages to and receive messages from different signal 


intervals at v. This will help show that the mating relation is symmetric 


Lemma D.2.20 Send-Receive Consistency: Consider any pair of neighbors u,v 
and any execution a of R|L. Let a; be a normal receive event at u from v and let am, 
be a normal receive event at v from u. Let a and ay be the send events corresponding 
to a; and a,, respectively. Then there 1s a SIGNAL, event between a; and a, iff there 


1s a SIGNAL, event between a; and ay. 
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Figure D.2: Send-Receive consistency: there is a signal between the receive and the send at u iff there is 


a signal between the corresponding send and receive at v. 


Proof: Assume that k > 7 as shown in the Figure D.2. The other cases are similar. 


Suppose there is a SIGNAL, event, say ay between a; and ay. By Claim D.2.14, 
s;.status(u) = on. Then by Claim D.2.8 there must be some p, j < p < k’, such 
that s,.status(v) = off. Thus since p > j, it must be that p > 1. Also p < k and 
k <m. Sop <m. Thus there is a state s, that occurs in the interval [s;,s,,] in 
which status(v) = off. Also we know by Claim D.2.14 that s,,.status(v) = on. Thus 


by Claim D.2.3, a SIGNAL, event must occur in the interval [s;, 5]. 


The reverse argument is slightly different. Suppose there is a SIGNAL, event, say 
am between a; and am. By the code, s;.status(v) = on. Thus by Claim D.2.18, applied 
tO $1, Gm: and Gm we know that there is some p such that p < k and s,.status(u) 4 on 
and such that s,.M,,, does not contain a L-message. We claim that j < p. If not not 
S$» must lie in the interval [s;,s;] and in this interval we know that there is always a 
y-message in M,, by Claim D.2.14. But this contradicts the fact that s,.M,, does 


not contain a -message. 


Thus s, occurs in the interval [s;,s,] and status(u) # on. Also we know by 
Claim D.2.14 that s,.status(v) = on Thus by Claim D.2.3, a SIGNAL, event must 


occur in the interval |[s;,s,]. I 


We now show the third part of the consistency property (see Definition 7.3.5). 


Lemma D.2.21 Successful Sending of Messages: Consider any pair of neighbors 
u,v and any execution a of R|L. Between any safe SENDMy.(m) event and a later 
FREEM,,, event, there is either a RECEIVEM,y,.(m) event or a SIGNAL, event. 
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Final Interval 


Figure D.3: Mating of Final Signal Intervals: Messages sent in a final interval can only be received in 


another final interval. 


Proof: Let us denote the SENDM,.(m) event by a; and the FREEM,,, event by ag. 
Clearly | < k. 


Thus, by the code status(v) = on in s,. Also by the code, freem,|u] = true in s, 


and hence by H, M,,, does not contain any L-message in S,. 


Next, consider a;. Since a; is safe, by definition there must an action a, = 
FREEM,.(m) such that h < i and such that there is no other SENDM,,,(*) action 
between a; and a;. Thus, by the code freem,|u] = true in s, and s;_,. Thus, by the 
code, we see that either s;.status(v) = off or m belongs to M,,, in s; (i.e., the message 
m is placed on the queue at v to send to u). But in the first case, we are done by 
Claim D.2.3, which tells us there must be a SIGNAL, event between s; and sx. 


So consider the second case where m belongs to M,,, in s; and s;.status(v) = on. 
But we know that in the later state s,, m does not belong to M,,. Now, from the 
code, the only two actions that could remove m from M,,, in the interval [s;,s,] are a 
RECEIVEM,,.(m) event or a RECEIVE,,,(ABORT, *) event. In the former case, we are 
done; so consider the latter case. Now, because s;.mode(v) = on, we know from A that 
there is no (ABORT, *) packet in s;.equeue,|u]. Thus there must have been an action 
a; = SEND,.(ABORT, *) in the interval |s;,s,]. From the code, s;.status(v) = off. The 
lemma now follows from Claim D.2.3, which tells us there must be a SIGNAL, event 


between s; and s,. ff 


We now show the fourth part of the consistency property (see Definition 7.3.5). 
The lemma on which it is based is sketched in Figure D.3. 
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Lemma D.2.22 Mating of Final Signal Intervals: Consider any pair of neighbors 
u,v and any execution a of R|L. Let a; be a normal receive event at u from v ina 
and a; be the send corresponding to a;. Then there is no SIGNAL, event after a; iff 
there there is no SIGNAL, event after aj. 


Proof: Suppose there is a SIGNAL, event a, after a. Then by Claim D.2.8, there is 
a state s,, between a; and a, such that s,,.mode(u) # Ready and sm.M,,, does not 
contain any ) messages. It follows from Claim D.2.14 that s,, must occur after a;. 
Thus there is a state after a; in which mode(u) #4 Ready. Thus by the Signal Lemma 
(Lemma D.2.2), a SIGNAL, event will occur after a;. 


The reverse argument is similar but slightly simpler. J 


Lemma D.2.23 Mating Relation Preserves Temporal Ordering: Consider any 
pair of neighbors u,v and any execution a of R\L. Suppose a signal interval S,, at u 
is mated to a signal interval S, at v and a signal interval S!, at u is mated to a signal 


interval Si at v. Then if Si, occurs later than S,, then S’ occurs later than S,. 


Proof: Omitted. Follows, in essence, from the FIFO properties of the underlying 
UDLs and the fact that ABORT packets sent between signal intervals flush the links 


and buffers of previously sent messages. J 


And now we come to the main result of this subsection: 
Lemma D.2.24 Every behavior of R\L satisfies the consistency property. 


Proof: We define a signal interval at a node wu in an execution a by analogy with 
the definitions for behaviors. We define two signal intervals [,, at wu and I, at v to be 
mates if any normal message received in [,, was sent in I, or vice versa. Lemma D.2.19 
and Lemma D.2.20 show that the mating relation is well-defined and that any signal 
interval at wu can have at most one mate and that the relation is symmetric. The last 
three conditions in Definition 7.3.5 follow from Lemma D.2.21, Lemma D.2.22 and 
Lemma D.2.23. J 
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D.2.4 Every Behavior of R|L is causal 


We can now prove that the Reset protocol is causal. We first prove the first property 
in Definition 7.3.6. 


Theorem D.2.25 Consider any execution a = 89,01,51,... of R|L. There is some 
constant c such that every signal event a, that occurs at time greater than s9.time+ cn 


as preceded by a request event a; such that a,.time — a;.time < cn. 


Proof: A formal proof can be patterned after the intuitive argument given in Section 7.7. 


Next, we prove the second property in Definition 7.3.6. 


Lemma D.2.26 Consider an execution a = 589,41,51,... of R|L. There is some 


constant c such that a SIGNAL, event occurs within cn time of any REQUEST, event. 
Proof: We know from the code that in the state immediately following a REQUEST, 


event, status(u) = off. The lemma follows immediately from the Signal Lemma 


(Lemma D.2.2). I 


And now we come to the main result of this subsection: 
Lemma D.2.27 Every behavior of R|L is causal. 


Proof: Immediate from Definition 7.3.6 and Lemmas D.2.25 and D.2.26. Jj 


D.2.5 The Main Theorem 


Finally, we can state our main theorem of this section, which follows from the main 


lemmas of the last three subsections: 
Theorem D.2.28 Every behavior of R|L is in RP. 


Proof: Immediate from Definition 7.3.7 and Lemmas D.2.17, D.2.24, and D.2.27. JJ 
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Appendix E 


Dijkstra’s Token Protocol as an 


Example of Counter Flushing 


In Chapter 10, we described a paradigm called counter flushing. We now show that Di- 


jkstra’s first example protocol in [Dij74] can be simply understood using this paradigm. 


Dijkstra’s first example is modelled by the automaton D2 shown in Figure E.1. 
As in the previous example, the nodes (once again numbered from 0 to n — 1) are 
arranged such that node 1 has node 0 and node 2 as its neighbors and so on. However, 
in this case we also assume that Process 0 and n — 1 are neighbors. In other words, 
by making 0 and n — 1 adjacent we have made the line into a ring. For process 1, let 
us call Process 1 — 1 (we assume that all arithmetic on indices and counters is mod n) 


the anticlockwise neighbor of 1 and z+ 1 the clockwise neighbor of 2. 


Each node has a counter count; in the range 0,...n that is incremented mod n+ 1. 
Once again the easiest way to understand this protocol is to understand what happens 
when it is properly initialized. Thus assume that initially Process 0 has its counter set 
to 1 while all other processes have their counter set to 0. Processes other than 0 are 
only allowed to “move” (see Figure E.1) when their counter differs in value from that 
of their anticlockwise neighbor; in this case, the process is allowed to make a move 
by setting its counter to equal that of its anticlockwise neighbor. Thus initially, only 
Process 1 can make a move after which Process 1 has its counter equal to 1; next, 
only Process 2 can move, after which Process 2 sets its counter equal to 1; and so on, 
until the value 1 moves clockwise around the ring until all processes have their counter 


equal to 1. 
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The state of the system consists of an integer variable 
count; € {0,...n}, one for every process in the ring. 


We assume that Process 0 and n — 1 are neighbors 
In the initial state count; — 0 fori =—1...n—1 and count; = 1 


MovEp (*action for Process 0 only *) 
Precondition: county = count,_1 (*equal to anticlockwise neighbor?*) 


Effect: county := (counto + 1) mod (n + 1) (*increment counter*) 


MoveE;,1 <i <n-—1 (*action for other processes*) 
Precondition: count; # count;_; (*not equal to anticlockwise neighbor?*) 
Effects: 
count; := count;_1;(*set equal to anticlockwise neighbor*) 


All actions are in a separate class 


Figure E.1: Automaton D1: a version of Dijkstra’s first example with initial states. The protocol does 


token passing on a ring using nodes with n states. 
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Process 0 


Figure E.2: In the good states for Dijktra’s first example, the ring can be partitioned into 2 bands with 
the token at the boundary. 


Process 0 on the other hand cannot make a move until Process n — 1 has the 
same counter value as Process 0. Thus until Process 1 sets its counter to 1, Process 0 
cannot make a move. However, when this happens, Process 1 increments its counter 
mod n+ 1. Then the cycle repeats as now the value 2 begins to move across the ring 
(assuming n > 2) and so on. Thus after proper initialization, this system does perform 
a form of token passing on a ring; each node is again considered to have the token, 


when the system is in a state in which the node can take a move. 


The good global states of the protocol can be sketched in Figure E.2. Notice that 
the ring can be partitioned into 2 bands. All counter values within a band are equal 
and the band that includes the top node has a counter value one higher than the lower 
band. The token is at the boundary between the two bands; after that node makes a 


move, the top band becomes larger and the lower band becomes smaller. 


It is easy to see that the system is in a good state iff the following local predicates 


are true. 


e Forti =1...n—1, either count;_,; = count; or count;_; = count; + 1. 


e Either county = count,_, or county = count,_; +1. 
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The system is locally checkable but it does not appear to be locally correctable. 
However, it does stabilize using a paradigm that we can call counter flushing. Even 
if the counter values are arbitrarily initialized (in the range 0,...,n) the system will 
eventually begin executing as some suffix of a properly initialized execution. We will 


prove this informally using three claims: 


e In any execution, Process 0 will eventually increment its counter. Sup- 
pose not. Then since Process 0 is the only process that can “produce” new 
counter values, the number of distinct counter values cannot increase. If there 
are two or more distinct counter values, then moves by Processes other than 0 
will reduce the number of distinct counter values to 1, after which Process 0 will 


increment its counter. 


e In any execution, Process 0 will eventually reach a “fresh” counter 
value that is not equal to the counter values of any other process. To 
see this, note that that in the initial state there are at most n distinct counter 
values. Thus there is some counter value say m that is not present in the initial 
state. Since, process 0 keeps incrementing its counter, Process 0 will eventually 


reach m and in the interim no other process can set their counter value to m. 


e Any state in which Process 0 has a fresh counter value m is eventually 
followed by a state in which all processes have counter value m. It is 
easy to see that the value m moves clockwise around the ring “flushing” any 
other counter values, while Process 0 remains at m. This is why we call this 


paradigm counter flushing. 


The net effect is that any execution of D1 eventually reaches a good state in which 
it remains. The reader should compare our proof of this protocol with the general 


description of counter flushing found in Chapter 10. 
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